This gradual approach looks good to me. And, if the additional compression algorithm like lz4 is always better than pglz for every scenarios, we can just change the code so that the additional algorithm is always used. Which would make the code simpler.
Right.
> 3. Compressing one block vs all blocks: > Andres suggested that compressing all backup blocks in one go may give us > better compression ratio. This is worth trying. I'm wondering what would the > best way to do so without minimal changes to the xlog insertion code. Today, > we add more rdata items for backup block header(s) and backup blocks > themselves (if there is a "hole" then 2 per backup block) beyond what the > caller has supplied. If we have to compress all the backup blocks together, > then one approach is to copy the backup block headers and the blocks to a > temp buffer, compress that and replace the rdata entries added previously > with a single rdata.
Basically sounds reasonable. But, how does this logic work if there are multiple rdata and only some of them are backup blocks?
My idea is to just make a pass over the rdata entries past the rdt_lastnormal element after processing the backup blocks and making additional entries in the chain. These additional rdata entries correspond to the backup blocks and their headers. So we can copy the rdata->data of these elements in a temp buffer and compress the entire thing in one go. We can then replace the rdata chain past the rdt_lastnormal with a single rdata with data pointing to the compressed data. Recovery code just needs to decompress this data the record header indicates that the backup data is compressed. Sure the exact mechanism to indicate if the data is compressed (and by which algorithm) can be worked out.
If a "hole" is not copied to that temp buffer, ISTM that we should change backup block header so that it contains the info for a "hole", e.g., location that a "hole" starts. No?
AFAICS its not required if we compress the stream of BkpBlock and the block data. The current mechanism of constructing the additional rdata chain items takes care of hole anyways.