Обсуждение: pg_dump custom format and pigz
Hello,
I'm using pg_dump intensively, and until now using the plaintext format which allows me to pipe the output to pigz (http://zlib.net/pigz/)
This is the fastest way I've found to generate compressed backups.
(These backups are very rarely used)
I would like to switch to the custom format in order to take advantage of the parallel restore enhancement, but this would force me to give pigz up, hence loosing performances with pg_dump.
Is there a way to combine the custom format with pigz?
feature request ?
best regards,
Marc Mamin
Hi Marc, On Fri, Oct 23, 2009 at 02:48:58PM +0200, Marc Mamin wrote: > I'm using pg_dump intensively, and until now using the plaintext format > which allows me to pipe the output to pigz (http://zlib.net/pigz/) > This is the fastest way I've found to generate compressed backups. > > (These backups are very rarely used) > > I would like to switch to the custom format in order to take advantage > of the parallel restore enhancement, but this would force me to give > pigz up, hence loosing performances with pg_dump. > Is there a way to combine the custom format with pigz? You might add pigz as a post-processing step and disabling compression in pg_dump. HTH, Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.tisc.de
Hello, > You might add pigz as a post-processing step and disabling compression in pg_dump. The problem with this solution is that it make it necessary to decompress the dump entirely before using pg_restore (or didI miss a point ?) This is an issue as our uncompressed dumps may reach 3-500 GB... A better way would be to have a configuration option to tell which compression tool should be used internally by pg_dump. best regards, Marc Mamin -----Original Message----- From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Tino Schwarze Sent: Friday, October 23, 2009 3:11 PM To: pgsql-admin@postgresql.org Subject: Re: [ADMIN] pg_dump custom format and pigz Hi Marc, On Fri, Oct 23, 2009 at 02:48:58PM +0200, Marc Mamin wrote: > I'm using pg_dump intensively, and until now using the plaintext > format which allows me to pipe the output to pigz > (http://zlib.net/pigz/) This is the fastest way I've found to generate compressed backups. > > (These backups are very rarely used) > > I would like to switch to the custom format in order to take advantage > of the parallel restore enhancement, but this would force me to give > pigz up, hence loosing performances with pg_dump. > Is there a way to combine the custom format with pigz? You might add pigz as a post-processing step and disabling compression in pg_dump. HTH, Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.tisc.de -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Hi Marc, On Fri, Oct 23, 2009 at 03:52:16PM +0200, Marc Mamin wrote: > > You might add pigz as a post-processing step and disabling compression in pg_dump. > > The problem with this solution is that it make it necessary to > decompress the dump entirely before using pg_restore (or did I miss a > point ?) > This is an issue as our uncompressed dumps may reach 3-500 GB... Right, that would not work. > A better way would be to have a configuration option to tell which > compression tool should be used internally by pg_dump. Well, that would mean linking pg_dump with the library providing the compression you want. I suppose, only data is compressed, not the whole dump, e.g. dumping without compression then compressing is different from dumping with compression. Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.tisc.de
>Well, that would mean linking pg_dump with the library providing the compression you want. I suppose, only data is compressed,not the whole dump, e.g. dumping without compression then compressing is different from dumping with compression. Yep, I have no skill in C programming, but wouldn't it be a good idea to add/integrate the pigz source in pg_dump ? There is aleast no concern to have about the License. regards, Marc -----Original Message----- From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Tino Schwarze Sent: Friday, October 23, 2009 4:25 PM To: pgsql-admin@postgresql.org Subject: Re: [ADMIN] pg_dump custom format and pigz Hi Marc, On Fri, Oct 23, 2009 at 03:52:16PM +0200, Marc Mamin wrote: > > You might add pigz as a post-processing step and disabling compression in pg_dump. > > The problem with this solution is that it make it necessary to > decompress the dump entirely before using pg_restore (or did I miss a > point ?) This is an issue as our uncompressed dumps may reach 3-500 > GB... Right, that would not work. > A better way would be to have a configuration option to tell which > compression tool should be used internally by pg_dump. Well, that would mean linking pg_dump with the library providing the compression you want. I suppose, only data is compressed,not the whole dump, e.g. dumping without compression then compressing is different from dumping with compression. Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.tisc.de -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin