Re: [FEATURE REQUEST] Streaming Onlinebackup (MaybeOFFTOPIC)

>>> On Fri, Sep 28, 2007 at  9:38 AM, in message
<46FCCB89.EE98.0025.0@wicourts.gov>, "Kevin Grittner"
<Kevin.Grittner@wicourts.gov> wrote:
>>>> On Fri, Sep 28, 2007 at  5:53 AM, in message
> <E1539E0ED7043848906A8FF995BDA57902685F43@m0143.s-mxs.net>, "Zeugswetter
> Andreas ADI SD" <Andreas.Zeugswetter@s-itsolutions.at> wrote:
>> archive_command=dd if=%p of=/backup/WAL/%f bs=1 count=%b
> I think
> your example would need to cat the dd you showed with one which drew
> from /dev/zero.  I'll run a few tests with full and nearly empty files
> using hand-generated values and see how the performance of this in
> front of gzip compares to the filter.
After Tom's email, this is pretty academic; but here are the results
for our "best case" example:
pg_clearxlogtail | gzip:
real    0m0.132s
user    0m0.119s
sys     0m0.024s
(dd if=00000001000000040000001A bs=1 count=132 ; dd if=/dev/zero bs=1 count=16777084) | gzip >
../kjgtest2/00000001000000040000001A.2.gz
132+0 records in
132+0 records out
16777084+0 records in
16777084+0 records out

real    0m19.243s
user    0m3.211s
sys     0m27.135s
That's a lot worse.  I switched the bs and count:
(dd if=00000001000000040000001A bs=132 count=1 ; dd if=/dev/zero bs=16777084 count=1) | gzip >
../kjgtest2/00000001000000040000001A.3.gz
1+0 records in
1+0 records out
1+0 records in
1+0 records out

real    0m0.196s
user    0m0.173s
sys     0m0.025s
The filter code still wins.
The "worst case" example:
pg_clearxlogtail | gzip:
real    0m1.073s
user    0m1.018s
sys     0m0.063s
gz size: 4554307
ADAMS-PG:/var/pgsql/data/kjgtest # time dd if=0000000100000003000000F0 bs=16777216 count=1 | gzip >
../kjgtest2/0000000100000003000000F0.3.gz
1+0 records in
1+0 records out
Marginal improvement.
real    0m1.001s
user    0m0.923s
sys     0m0.081s
ADAMS-PG:/var/pgsql/data/kjgtest # time cat 0000000100000003000000F0 | gzip > ../kjgtest2/0000000100000003000000F0.4.gz

real    0m1.109s
user    0m1.055s
sys     0m0.062s
Not quite as good.  Since the archiver process can't actually deliver
this number in a lightweight manner, all it goes to show is that the
filter code compares reasonably well in performance with dd and cat.
-Kevin