Re: TDE: Benchmarking WAL encryption approaches

От:

Zsolt Parragi <zsolt.parragi@percona.com>

Дата:

20 июля в 09:47:41

> You did share a benchmark result for the normal transactions, but these
> approaches can also affect the recovery time and replication, right? Might be
> better to do as well.

I didn't share them in my previous email because for two reasons:
* I didn't run any read tests before writing that email
* my expectation was that I'll have to do some optimizations before I
get good read results, and I didn't want to delay sharing my initial
benchmarks because of that.

I've been only trying to measure reads for a few days now, so I do not
want to say that I have final proper results, but based on what I see
so far: that assumption was wrong, recovery/replication performance in
quite good.

My test approach is to:
* generate a large amount of WAL without checkpoints
* kill the server
* take a snapshot, measure recovery speed, restore snapshot, repeat

And then I do the same thing with replica catchup time, and I also
tried to measure if I see any difference in replication lag. In all
time measurements, I also logged user/sys times separately. This makes
it clear that in some scenarios the overall time remains the same, but
the user/sys ratio shifts.

I repeated this with three workloads, one using bulk records (e.g.
huge INSERT/COPY commands), another focusing on many small wal records
(small OLTP writes) and a third one inserting mostly full page writes.

All with the usual benchmarking precautions (dropping caches, setting
cpu governor and other settings. without dropping caches I only see
noise).

Replication shows increased user CPU usage, which is more significant
in the perpage version where the current implementantion decrypts on
the primary/encrypts again on the replica. In the perrecord version,
it only has to decrypt the wal records to apply them. But this doesn't
show up in replication performance, I didn't see any increased
replication lag compared to master in any of my tests so far.

Replica catchup shows a ~20% increased user CPU time across all
workloads, and the time impact varies across workloads, but there's no
significant difference between perrecord and perpage. Full page writes
are the worst with ~12% slower catchup time, bulk writes need 5% more
time and for OLTP encryption is invisible.

Recovery shows the same 20% CPU time, just with different overall
additional time numbers: FP 13%, bulk 12%, OLTP same as not encrypted,
only the USR CPU difference is there.

TDE: Benchmarking WAL encryption approaches

От:

Zsolt Parragi <zsolt.parragi@percona.com>

Дата:

16 июля в 20:21:27

Hello!

During the TDE discussions at pgconf.dev, I promised some benchmarks
about performance characteristics regarding different WAL encryption
options, to provide solid data before deciding on an approach.

I ended up benchmarking and optimizing different variants more than I
initially planned, but now I have some conclusions that I think are
interesting to share.

I will only focus on two variants that make sense from a performance
perspective, per page or per record AES-CTR-256. Other encryption
modes / integrations all result in significantly decreased performance
(or no visible benefit for added complexity). For example AES-GCM can
be implemented in both versions, but because GCM is slower, the end
result will be slower. I can also share numbers about GCM/XTS and
other possibilities if people are interested, but for a "simple
implementation in the core", I think it should be safe to assume that
we want to go with CTR.

So the two approaches are:

1. ctr_pagelevel: when we flush or read a wal page, we encrypt/decrypt
it. This is simple, encrypts absolutely everything, but does
everything in the critical section holding the WAL lock.
2. ctr_perrecord: encryption happens when we write a WAL record. Each
record has to hold an additional IV, since at this point we can't rely
on the LSN. It parallelizes encryption at the cost of increased
complexity in the backend. Only the record data is encrypted, headers
stay plaintext, which slightly simplifies command-line tools.

I attached (one version) of both patches. All patches I used for
testing are minimal: encryption is hardcoded, uses a hardcoded key
baked into the source, there are no options to disable it or change
settings, only the minimum required for performance testing and
validation (making sure that we indeed encrypt the WAL).

The naive implementation of perrecord is clearly slower than
plaintext/pagelevel, however, after optimizing both versions, we can
see the advantages of it: pagelevel performance degrades in scenarios
with many clients and high WAL throughput, as the lock contention
increases.

In my tests, I focused on write performance. Read performance
(recovery, replication) is a different topic, and my implementations
for both could be optimized better - I didn't try to measure and
improve them yet.

I used 3 test scenarios:

* sysbench oltp_read_write
* sysbench oltp_write_only
* A scenario in which each session repeats the following:
  1. select 100 random IDs (locally, no SQL queries, as ids are serial
and continuous)
  2. execute an UPDATE SET c=c+1 WHERE id IN (id list);

For testing, I used an AMD Threadripper 3970X (32 cores) and an Intel
Optane P5800X for storage, which doesn't have the same throttling
limitations as consumer SSDs, the tests weren't limited by IO.

In the read_write scenario, there's no difference between the 2
encryption implementation and unencrypted master at all. There's some
noise, but we can't notice the effect of encryption at all.

In the write_only scenario, there's still no difference between master
and ctr_perrecord, but ctr_pagelevel starts to fall behind at 32/64
threads. The difference disappears at 128 threads, as at that point
master performance also degrades in my test setup.

In the update_inlist scenario which increases WAL throughput more
quickly, it already falls behind the other two at 8 threads, and
remains there for all following runs: everywhere except in the single
threaded case we can see a clear 20% performance drop. Similarly to
the write_only scenario, there's no measurable difference between
master and ctr_perrecord.

This is in line with my earlier observations during less detailed
benchmarking, that if we want to guarantee good WAL encryption
performance, we have to go with per record encryption. The results
also depend on the exact hardware used for the tests, scenario, and so
on, but I think the general point of it is clear: the more work we do
in the critical section, the worse the performance loss can be in
certain configurations. Going with a choice that adds no additional
workload there seems like a generally safer choice to me.

Please share any feedback / improvement ideas for the patches, or
different approaches we could test, I can continue benchmarking more
scenarios / prototypes if anybody has ideas. Unless there's a
different conclusion, I'll proceed with sharing a first prototype of a
TDE proposal using the per record approach in the following weeks.

I also attached the actual results of a comparison run as a CSV file.Hello!

During the TDE discussions at pgconf.dev, I promised some benchmarks
about performance characteristics regarding different WAL encryption
options, to provide solid data before deciding on an approach.

I ended up benchmarking and optimizing different variants a lot more
than I initially planned, but now I have some conclusions that I think
are interesting to share.

I will only focus on two variants that make sense from a performance
perspective, per page or per record AES-CTR-256. Other encryption
modes / integrations all result in significantly decreased performance
(or no visible benefit for added complexity). For example AES-GCM can
be implemented in both versions, but because GCM is slower, the end
result will be slower. I can also share numbers about GCM/XTR and
other possibilities if people are interested, but for a "simple
implementation in the core", I think it should be safe to assume that
we want to go with CTR.

So the two approaches are:

1. ctr_pagelevel: when we flush or read a wal page, we encrypt/decrypt
it. This is simple, encrypts absolutely everything, but does
everything in the critical section holding the WAL lock.
2. ctr_perrecord: encryption happens when we write a WAL record. Each
record has to hold an additional IV, since at this point we can't rely
on the LSN. Parallelizes encryption, at the cost of increasing
complexity. Only the record data is encrypted, headers stay plaintext.

I attached (one version) of both patches. All patches I used for
testing are minimal: encryption is hardcoded, uses a hardcoded key
baked into the source, there are no options to disable it or change
settings, only the minimum required for performance testing and
validation (making sure that we indeed encrypt the WAL).

The naive implementation of perrecord is clearly slower than
plaintext/pagelevel, however, after optimizing both versions, we can
see the advantages of it: pagelevel performance degrades in scenarios
with many clients and high WAL throughput, as the lock contention
increases.

In my tests, I focused on write performance. Read performance
(recovery, replication) is a different topic, and my implementations
for both could be optimized better - I didn't try to measure and
improve them yet.
I used 3 test scenarios:

* sysbench oltp_read_write
* sysbench oltp_write_only
* A scenario in which each session repeats the following:
  1. select 100 random IDs (locally, no SQL queries, as ids are serial
and continuous)
  2. execute an UPDATE SET c=c+1 WHERE id IN (id list);

For testing, I used an AMD Threadripper 3970X (32 cores) and an Intel
Optane P5800X for storage, which doesn't have the same throttling
limitations as consumer SSDs, the tests weren't limited by IO.

In the read_write scenario, there's no difference between the 2
encryption implementation and unencrypted master at all. There's some
noise, but we can't notice the effect of encryption at all.

In the write_only scenario, there's still no difference between master
and ctr_perrecord, but ctr_pagelevel starts to fall behind at 32/64
threads. The difference disappears at 128 threads, as at that point
master performance also degrades in my test setup.

In the update_inlist scenario which increases WAL throughput more
quickly, it already falls behind the other two at 8 threads, and
remains there for all following runs: everywhere except in the single
threaded case we can see a clear 20% performance drop. Similarly to
the write_only scenario, there's no measurable difference between
master and ctr_perrecord.

This is in line with my earlier observations during less detailed
benchmarking, that if we want to guarantee good WAL encryption
performance, we have to go with per record encryption. The results
also depend on the exact hardware used for the tests, scenario, and so
on, but I think the general point of it is clear: the more work we do
in the critical section, the worse the performance loss can be in
certain configurations. Going with a choice that adds no additional
workload there seems like a generally safer choice to me.

Please share any feedback / improvement ideas for the patches, or
different approaches we could test, I can continue benchmarking more
scenarios / prototypes if anybody has ideas. Unless there's a
different conclusion, I'll proceed with sharing a first prototype of a
TDE proposal using the per record approach in the following weeks.

I also attached the actual results of a comparison run as a CSV file.

RE: TDE: Benchmarking WAL encryption approaches

От:

"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>

Дата:

17 июля в 16:30:06

Dear Zsolt,

Thanks for sharing the good benchmark!

> Please share any feedback / improvement ideas for the patches, or
> different approaches we could test, I can continue benchmarking more
> scenarios / prototypes if anybody has ideas. Unless there's a
> different conclusion, I'll proceed with sharing a first prototype of a
> TDE proposal using the per record approach in the following weeks.

You did share a benchmark result for the normal transactions, but these
approaches can also affect the recovery time and replication, right? Might be
better to do as well.

Best regards,
Hayato Kuroda
FUJITSU LIMITED