Greg, I think you might be onto something here. your suggestion prompts
to get rid off --remove-sent-files in my rsync command, and clean up wal
files from master by some other means, this way even if master's rsync
picks partial file on the first run, it should update and complete file
once it's finished being copied by archive_command to my buffer directory.
Thanks! I'll get this in asap, if this does not solve the problem I'll
re-post
V.
Greg Smith wrote:
> On Sat, 17 May 2008, Ioannis Tambouras wrote:
>
>> The archive command tests if the wal segment exists and is a file,
>> but it does not check if the file is still being written.
>
> That's because it doesn't have to; the archive command doesn't get
> called until the writing is done.
>
>> I don't have sources of pg_standby near me, but I remember in the
>> C code checks for complete segment sizes.
>
> That's on the receiving side, to make sure it's not trying to process
> files that haven't finished copying to the standby yet. You don't
> have to do any of that in the archive_command.
>
> Anyway, back to the original question:
>
>> archive_command = 'test ! -f /usr/local/wal_archive_local/%f && cp %p
>> /usr/local/wal_archive_local/%f'
>> archive files are then moved on master to standby every other minute:
>> rsync -aq --remove-sent-files /usr/local/wal_archive_local/
>> slave::wal_archive/
>
> I don't see any mechanism here to keep rsync from copying over partial
> files to the standby before they've finished copying to the
> wal_archive_local directory. That's my guess for where the small
> files are coming from, rsync before the cp is done. If you're going
> to buffer in a transfer directory, you need some sort of test or
> locking to make sure the file is complete with exactly 16MB before it
> gets rsync'd over. I suspect no amount of poking at the standby will
> root out the issue because it's happening on the primary.
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
--
________________________________________
Vladimir (Vlad) Kosilov
Senior Systems Administrator
Contigo Systems Inc.
604.683.3106 (phone)
604.648.9886 (fax)
vkosilov@contigo.com
www.contigo.com
________________________________________