Re: libpq compression

On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote:
> On Jun25, 2012, at 04:04 , Robert Haas wrote:
> > If, for
> > example, someone can demonstrate that an awesomebsdlz compresses 10x
> > as fast as OpenSSL...  that'd be pretty compelling.
> 
> That, actually, is demonstrably the case for at least Google's snappy.
> (and LZO, but that's not an option since its license is GPL) They state in
> their documentation that
> 
>   In our tests, Snappy usually is faster than algorithms in the same class
>   (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable
>   compression ratios.
> 
> The only widely supported compression method for SSL seems to be DEFLATE,
> which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib
> a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by
> an order of magnitude.
> 
> The compression ratio achieved by DEFLATE/gzip/zlib is much better, though.
> The snappy documentation states
> 
>   Typical compression ratios (based on the benchmark suite) are about
>   1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for
>   JPEGs, PNGs and other already-compressed data. Similar numbers for zlib
>   in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively.
> 
> Here are a few numbers for LZO vs. gzip. Snappy should be comparable to
> LZO - I tested LZO because I still had the command-line compressor lzop
> lying around on my machine, whereas I'd have needed to download and compile
> snappy first.
> 
> $ dd if=/dev/random of=data bs=1m count=128
> $ time gzip -1 < data > data.gz
> real    0m6.189s
> user    0m5.947s
> sys    0m0.224s
> $ time lzop < data > data.lzo
> real    0m2.697s
> user    0m0.295s
> sys    0m0.224s
> $ ls -lh data*
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:43 data
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:44 data.gz
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:44 data.lzo
> 
> $ dd if=/dev/zero of=zeros bs=1m count=128
> $ time gzip -1 < zeros > zeros.gz
> real    0m1.083s
> user    0m1.019s
> sys    0m0.052s
> $ time lzop < zeros > zeros.lzo
> real    0m0.186s
> user    0m0.123s
> sys    0m0.053s
> $ ls -lh zeros*
> -rw-r--r--  1 fgp  staff   128M Jun 25 14:47 zeros
> -rw-r--r--  1 fgp  staff   572K Jun 25 14:47 zeros.gz
> -rw-r--r--  1 fgp  staff   598K Jun 25 14:47 zeros.lzo
> 
> To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about
> 350MB/s if the data is purely random, and about 800MB/s if the data
> compresses extremely well. (Numbers based on user time since that indicates
> the CPU time used, and ignores the IO overhead, which is substantial)
> 
> IMHO, the only compelling argument (and a very compelling one) to use
> SSL compression was that it requires very little code on our side. We've
> since discovered that it's not actually that simple, at least if we want
> to support compression without authentication or encryption, and don't
> want to restrict ourselves to using OpenSSL forever. So unless we give
> up at least one of those requirements, the arguments for using
> SSL-compression are rather thin, I think.
> 
> best regards,
> Florian Pflug
> 
+1 for http://code.google.com/p/lz4/ support. It has a BSD license too.
Using SSL libraries give all the complexity without any real benefit.

Regards,
Ken