Re: libpq compression

Hi!

> On 09.02.2021 09:06, Konstantin Knizhnik wrote:
>
> Sorry, but my interpretation of your results is completely different:
> permanent compression is faster than chunked compression (2m15 vs. 2m27)
> and consumes less CPU (44 vs 48 sec).
> Size of RX data is slightly larger - 0.5Mb but TX size is smaller - 5Mb.
> So permanent compression is better from all points of view: it is
> faster, consumes less CPU and reduces network traffic!
>
> From my point of view your results just prove my original opinion that
> possibility to control compression on the fly and use different
> compression algorithm for TX/RX data
> just complicates implementation and given no significant advantages.

When I mentioned the lower CPU usage, I was referring to the pgbench test results in attached
google doc, where chunked compression demonstrated lower CPU usage compared to the permanent compression.

I made another (a little bit larger) pgbench test to demonstrate this:

Pgbench test parameters:

Data load
pgbench -i -s 100

Run configuration
pgbench --builtin tpcb-like -t 1500 --jobs=64 --client==600"

Pgbench test results:

No compression
latency average = 247.793 ms
tps = 2421.380067 (including connections establishing)
tps = 2421.660942 (excluding connections establishing)

real    6m11.818s
user    1m0.620s
sys     2m41.087s
RX bytes diff, human: 703.9221M
TX bytes diff, human: 772.2580M

Chunked compression (compress only CopyData and DataRow messages)
latency average = 249.123 ms
tps = 2408.451724 (including connections establishing)
tps = 2408.719578 (excluding connections establishing)

real    6m13.819s
user    1m18.800s
sys     2m39.941s
RX bytes diff, human: 707.3872M
TX bytes diff, human: 772.1594M

Permanent compression
latency average = 250.312 ms
tps = 2397.005945 (including connections establishing)
tps = 2397.279338 (excluding connections establishing)

real    6m15.657s
user    1m54.281s
sys     2m37.187s
RX bytes diff, human: 610.6932M
TX bytes diff, human: 513.2225M


As you can see in the above results, user CPU time (1m18.800s vs 1m54.281s) is significantly smaller in
chunked compression because it doesn’t try to compress all of the packets.

Here is the summary from my POV, according to these and previous tests results:

1. Permanent compression always brings the highest compression ratio
2. Permanent compression might be not worthwhile in load different from COPY data / Replication / BLOBs/JSON queries
3. Chunked compression allows to compress only well compressible messages and save the CPU cycles by not compressing
theothers 
4. Chunked compression introduces some traffic overhead compared to the permanent (1.2810G vs 1.2761G TX data on
pg_restoreof IMDB database dump, according to results in my previous message) 
5. From the protocol point of view, chunked compression seems a little bit more flexible:
 - we can inject some uncompressed messages at any time without the need to decompress/compress the compressed data
 - we can potentially switch the compression algorithm at any time (but I think that this might be over-engineering)

Given the summary above, I think it’s time to make a decision on which path we should take and make the final list of
goalsthat need to be reached in this patch to make it committable.  

Thanks,

Daniil Zakhlystov