Re: Improving compressibility of WAL files

On Thu, 2009-01-08 at 18:02 -0500, Aidan Van Dyk wrote:
> * Bruce Momjian <bruce@momjian.us> [090108 16:43]:
> > 
> > The attached patch from Aidan Van Dyk zeros out the end of WAL files to
> > improve their compressibility. (The patch was originally sent to
> > 'general' which explains why it was lost until now.)
> > 
> > Would someone please eyeball it?;  it is useful for compressing PITR
> > logs even if we find a better solution for replication streaming?
> 
> The reason I didn't push it was that people claimed it would chew up to
> much WAL bandwidhh (causing a large commit latency) when the new 0's are
> all written/fsynced at once...
> 
> I don't necessarily buy it, because the force_switch is usually either a
> 1) timeed occurance on an otherwise idle time
> 2) user-forced (i.e. forced checkpoint/pg_backup, so your IO is going to
>    be hammered anyways...
> 
> But that's why I didn't follow up on it...
> 
> There's possible a few other ways to do it, such as zero the WAL on
> recycling (but not fsyncing it), and hopefully most of the zero's get
> trickled out by the OS before it comes down to a single 16MB fsync, but
> not many people seemed too enthused about the whole WAL compressablitly
> subject...
> 
> But, the way I see things going on -hackers, I must admit, sync-rep (WAL
> streaming) looks like it's a long way off and possibly not even going to
> do what I want, so *I* would really like this wal zero'ing...
> 
> If anybody has any specific things with the patch ehty think needs
> chaning, I'll try and accomidate, but I do note that I never
> submitted it for the Commitfest...

won't it still be easier/less intrusive on inline core functionality and
more flexible to just record end-of-valid-wal somewhere and then let the
compressor discard the invalid part when compressing and recreate it
with zeros on decompression ?

-------------------
Hannu