Обсуждение: Question about wal_compression and what to expect

Поиск
Список
Период
Сортировка

Question about wal_compression and what to expect

От
Sean O'Grady
Дата:
Hello,

I've enabled wal_compression however I do not see any change in the 16MB size for each WAL file. Through all my googling and searching the mailing lists, I could not find a description if a change *should* be expected even though many of the articles I've read indicate compression can save disk space. 

I am currently thinking that the records in the WAL file are compressed for the purpose of writing less data to disk, but that the file is still padded to 16MB. If so, does that mean to realize a space savings for archiving that my archive_command should compress the file (gzip, bz2, etc)?

Thanks,
Sean O'Grady

Question about wal_compression and what to expect

От
"Wetmore, Matthew (CTR)"
Дата:

From my understanding,

 

Wal_compression will compress full page writes in the wal file.  The size of your wal is configurable.

 

So for your 16MB wal, it’s already compressed into the wal, and your wal_size won’t change.  (just more data can fit into your 16MB)

 

As far as archiving, I create a dir, then tar.gz the dir for storage. (if that was your question)

 

 

 

From: Sean O'Grady <sean@thingee.com>
Sent: Tuesday, June 20, 2023 11:19 AM
To: pgsql-admin@lists.postgresql.org
Subject: [EXTERNAL] Question about wal_compression and what to expect

 

Hello,

I've enabled wal_compression however I do not see any change in the 16MB size for each WAL file. Through all my googling and searching the mailing lists, I could not find a description if a change *should* be expected even though many of the articles I've read indicate compression can save disk space. 

I am currently thinking that the records in the WAL file are compressed for the purpose of writing less data to disk, but that the file is still padded to 16MB. If so, does that mean to realize a space savings for archiving that my archive_command should compress the file (gzip, bz2, etc)?

Thanks,
Sean O'Grady

Re: Question about wal_compression and what to expect

От
Sean O'Grady
Дата:
Yes, my question was essentially do I need to separately compress WAL files for storage/archiving even with wal_compression set. So, I think you've answered my question.

Thanks!
Sean

On June 20, 2023, "Wetmore, Matthew (CTR)" <matthew.wetmore@express-scripts.com> wrote:

From my understanding,

 

Wal_compression will compress full page writes in the wal file.  The size of your wal is configurable.

 

So for your 16MB wal, it’s already compressed into the wal, and your wal_size won’t change.  (just more data can fit into your 16MB)

 

As far as archiving, I create a dir, then tar.gz the dir for storage. (if that was your question)

 

 

 

From: Sean O'Grady <sean@thingee.com>
Sent: Tuesday, June 20, 2023 11:19 AM
To: pgsql-admin@lists.postgresql.org
Subject: [EXTERNAL] Question about wal_compression and what to expect

 

Hello,

I've enabled wal_compression however I do not see any change in the 16MB size for each WAL file. Through all my googling and searching the mailing lists, I could not find a description if a change *should* be expected even though many of the articles I've read indicate compression can save disk space. 

I am currently thinking that the records in the WAL file are compressed for the purpose of writing less data to disk, but that the file is still padded to 16MB. If so, does that mean to realize a space savings for archiving that my archive_command should compress the file (gzip, bz2, etc)?

Thanks,
Sean O'Grady

Re: Question about wal_compression and what to expect

От
Scott Ribe
Дата:
> On Jun 21, 2023, at 7:51 AM, Sean O'Grady <sean@thingee.com> wrote:
>
> Yes, my question was essentially do I need to separately compress WAL files for storage/archiving even with
wal_compressionset. So, I think you've answered my question. 

If WAL files use per-page compression internally, you won't get much benefit from compressing them. If they use lz4
internally,then sure, bzip'ing them will get a little benefit, but nowhere near the normal amount.

Re: Question about wal_compression and what to expect

От
Sean O'Grady
Дата:
With archive_timeout set I think WAL files will often (at least in my case) be created before they are filled, and in those cases there should be a larger benefit, correct?

Thanks


On June 21, 2023, Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Jun 21, 2023, at 7:51 AM, Sean O'Grady <sean@thingee.com> wrote:
> 
> Yes, my question was essentially do I need to separately compress WAL files for storage/archiving even with wal_compression set. So, I think you've answered my question.

If WAL files use per-page compression internally, you won't get much benefit from compressing them. If they use lz4 internally, then sure, bzip'ing them will get a little benefit, but nowhere near the normal amount.

Re: Question about wal_compression and what to expect

От
Scott Ribe
Дата:
> On Jun 21, 2023, at 8:04 AM, Sean O'Grady <sean@thingee.com> wrote:
>
> With archive_timeout set I think WAL files will often (at least in my case) be created before they are filled, and in
thosecases there should be a larger benefit, correct? 

Good point--I believe you are correct about that.

Re: Question about wal_compression and what to expect

От
Jeff Janes
Дата:
On Wed, Jun 21, 2023 at 9:51 AM Sean O'Grady <sean@thingee.com> wrote:
Yes, my question was essentially do I need to separately compress WAL files for storage/archiving even with wal_compression set. So, I think you've answered my question.

The internal compression has to compress each record in isolation, which is not nearly as good as compressing the entire file as there are patterns that exist between records not just within records.  So yes, you can still usefully compress the files for archival.  But if that is your main goal, then maybe you should turn off wal_compression, as it actually interferes with the whole-file compression.  Compressing just once at the whole-file level gives me better overall compression than compression both internally and then whole-file.

You should test this for yourself in your own data, as compressibility is not uniform over all work loads.  But you need to make sure you do the test correctly, compressing the same amount of "work" for each set-up.  When you have wal_compression on, more "work" should fit into each WAL file, so just compressing the same number of WAL files will not give you a fair comparison.

Cheers,

Jeff

Re: Question about wal_compression and what to expect

От
Sean O'Grady
Дата:
The internal compression has to compress each record in isolation, which is not nearly as good as compressing the entire file as there are patterns that exist between records not just within records.

I see, that's makes sense. My sense is that wal_compression is really first to alleviate disk IO pressure and secondarily as a space saving benefit. Since I've set about archive_timeout because of a relatively light workload, I probably shouldn't need to worry about IO issues...

Thanks,
Sean