Re: CRC algorithm (was Re: [REVIEW] Re: Compression of full-page-writes)

On 2014-09-12 23:03:00 +0300, Heikki Linnakangas wrote:
> On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote:
> >At 2014-09-12 22:38:01 +0300, hlinnakangas@vmware.com wrote:
> >>
> >>We probably should consider switching to a faster CRC algorithm again,
> >>regardless of what we do with compression.
> >
> >As it happens, I'm already working on resurrecting a patch that Andres
> >posted in 2010 to switch to zlib's faster CRC implementation.
> 
> As it happens, I also wrote an implementation of Slice-by-4 the other day
> :-). Haven't gotten around to post it, but here it is.
> 
> What algorithm does zlib use for CRC calculation?

Also slice-by-4, with a manually unrolled loop doing 32bytes at once, using
individual slice-by-4's. IIRC I tried and removing that slowed things
down overall. What it also did was move crc to a function. I'm not sure
why I did it that way, but it really might be beneficial - if you look
at profiles today there's sometimes icache/decoding stalls...

Hm. Let me look:
http://archives.postgresql.org/message-id/201005202227.49990.andres%40anarazel.de

Ick, there's quite some debugging leftovers ;)

I think it might be a good idea to also switch the polynom at the same
time. I really really think we should, when the hardware supports, use
the polynom that's available in SSE4.2. It has similar properties, can
implemented in software just the same...

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services