Re: warm standby with WAL shipping

Thank you Greg for taking the time to explain this as throughly as you
have.  I have found a logic problem in my code.  I still don't know if
we will use pg_standby as the wrapper code in PITRTools is python and we
are not a python shop.  Kinda want to stick with what we know (C, perl,
shell).  I'm certainly looking at rsync rather then scp, which really
makes more sense.

Greg Smith wrote:
> On Tue, 2 Jun 2009, Geoffrey wrote:
>
>> The problem with my current process is as noted, my script keeps
>> looking for the *.history file, but never sees it.
>
>> From the restore_command section of the documentation:
>
> "The command will be asked for log files that are not present in the
> archive; it must return nonzero when so asked. This is not an error
> condition."
>
> So if you're asked for a .history file, and you don't have one, return
> an error state saying as much and the right thing will happen--recovery
> continues.  More comments about the path everyone wanders down when
> trying to build their own tools here are at
> http://archives.postgresql.org/sydpug/2006-10/msg00001.php , you'll
> probably get some more insight into the details here reading that early
> commentary.
>
> But you still want to know where they might come from, right?  Those
> history files show up when you've started your backup server after
> recovering files from the original system.  You need to bring the backup
> system out of standby before you'll see one.  That results in a new
> timeline:
> http://www.postgresql.org/docs/8.3/static/continuous-archiving.html#BACKUP-TIMELINES
>
>
> Think about for a second:  if the original server is still running, but
> you've started the standby system too, there are two separate histories
> with a common ancestor possible.  One history has the original data plus
> what happened afterwards on the master, the other has the originals plus
> what happened afterwards on the standby, after it was started.  The fun
> part is that you can return to copying files from the master again, so
> that you've got both sets of files available.  You then choose which
> history to follow by adjusting the recovery_target_timeline parameter in
> the recovery.conf file.
>
> Anyway, while getting your hands dirty so you understand what's
> happening is a good idea, trying to fully reinvent pg_standby is an
> exercise destined to have a whole stack of little issues like these.
> Don't do that; it's taken years to get that code as mature as it is, and
> while you'll progress faster because you can stare at its source it will
> still take you a while.  Returning to your original motivation for doing
> that, I threw a suggestion for how to combine pg_standby with using scp
> as the transport mechanism into
> http://wiki.postgresql.org/wiki/Warm_Standby , you just need to buffer
> transfers into a holding area to get around the atomic copy issues
> here.  This requires using a non-trivial archive_command process though,
> you'll need to call a real script there to handle the multiple steps
> involved rather than just getting away with a one-line command for that
> setting.  I reinvent that wheel periodically for sites that can't or
> won't install rsync for the job instead (always some variant on "for
> security reasons").  Unfortunately those sites also don't like releasing
> the resulting code to the world at large, so I don't have a full sample
> to show you.
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>


--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin