Re: Compression of full-page-writes

Hello All,

0001-CompressBackupBlock_snappy_lz4_pglz extends patch on compression of
full page writes to include LZ4 and Snappy . Changes include making
"compress_backup_block" GUC from boolean to enum. Value of the GUC can be
OFF, pglz, snappy or lz4 which can be used to turn off compression or set
the desired compression algorithm.

0002-Support_snappy_lz4 adds support for LZ4 and Snappy in PostgreSQL. It
uses Andres’s patch for getting Makefiles working and has a few wrappers to
make the function calls to LZ4 and Snappy compression functions and handle
varlena datatypes.
Patch Courtesy: Pavan Deolasee

These patches serve as a way to test various compression algorithms. These
are WIP yet. They don’t support changing compression algorithms on standby .
Also, compress_backup_block GUC needs to be merged with full_page_writes.
The patch uses LZ4 high compression(HC) variant.
I have conducted initial tests which I would like to share and solicit
feedback

Tests use JDBC runner TPC-C benchmark to measure the amount of WAL
compression ,tps and response time in each of the scenarios viz .
Compression = OFF , pglz, LZ4 , snappy ,FPW=off

Server specifications:
Processors:Intel® Xeon ® Processor E5-2650 (2 GHz, 8C/16T, 20 MB) * 2 nos
RAM: 32GB
Disk : HDD    450GB 10K Hot Plug 2.5-inch SAS HDD * 8 nos
1 x 450 GB SAS HDD, 2.5-inch, 6Gb/s, 10,000 rpm


Benchmark:
Scale : 100
Command  :java JR  /home/postgres/jdbcrunner-1.2/scripts/tpcc.js  -sleepTime
600,350,300,250,250
Warmup time          : 1 sec
Measurement time     : 900 sec
Number of tx types   : 5
Number of agents     : 16
Connection pool size : 16
Statement cache size : 40
Auto commit          : false
Sleep time           : 600,350,300,250,250 msec

Checkpoint segments:1024
Checkpoint timeout:5 mins


Scenario           WAL generated(bytes)                   Compression
(bytes)       TPS (tx1,tx2,tx3,tx4,tx5)
No_compress      2220787088 (~2221MB)                 NULL
13.3,13.3,1.3,1.3,1.3 tps
Pglz                  1796213760 (~1796MB)                 424573328
(19.11%)     13.1,13.1,1.3,1.3,1.3 tps
Snappy             1724171112 (~1724MB)                 496615976( 22.36%)
13.2,13.2,1.3,1.3,1.3 tps
LZ4(HC)            1658941328 (~1659MB)                 561845760(25.29%)
13.2,13.2,1.3,1.3,1.3 tps
FPW(off)           139384320(~139 MB)                    NULL
13.3,13.3,1.3,1.3,1.3 tps

As per measurement results, WAL reduction using LZ4 is close to 25% which
shows 6 percent increase in WAL reduction when compared to pglz . WAL
reduction in snappy is close to 22 % .
The numbers for compression using LZ4 and Snappy doesn’t seem to be very
high as compared to pglz for given workload. This can be due to
in-compressible nature of the TPC-C data which contains random strings

Compression does not have bad impact on the response time. In fact, response
times for Snappy, LZ4 are much better than no compression with almost ½ to
1/3 of the response times of no-compression(FPW=on) and FPW = off.
The response time order for each  type of compression is
Pglz>Snappy>LZ4

Scenario              Response time (tx1,tx2,tx3,tx4,tx5)
no_compress        5555,1848,4221,6791,5747 msec
pglz                    4275,2659,1828,4025,3326 msec
Snappy               3790,2828,2186,1284,1120 msec
LZ4(hC)              2519,2449,1158,2066,2065 msec
FPW(off)             6234,2430,3017,5417,5885 msec

LZ4 and Snappy are almost at par with each other in terms of response time
as average response times of five types of transactions remains almost same
for both.
0001-CompressBackupBlock_snappy_lz4_pglz.patch
<http://postgresql.1045698.n5.nabble.com/file/n5805044/0001-CompressBackupBlock_snappy_lz4_pglz.patch>
0002-Support_snappy_lz4.patch
<http://postgresql.1045698.n5.nabble.com/file/n5805044/0002-Support_snappy_lz4.patch>




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Compression-of-full-page-writes-tp5769039p5805044.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.