Re: Sync Rep: First Thoughts on Code

On Thu, 2008-12-11 at 11:29 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Thu, 2008-12-11 at 09:44 +0200, Heikki Linnakangas wrote:
> >> Simon Riggs wrote:
> >>> When the WAL starts streaming the *primary* can immediately perform
> >>> synchronous replication, i.e. commit waits for transfer. 
> >> Until the standby has obtained all the missing log files, it's not 
> >> up-to-date, and there's no guarantee that it can finish the replay. For 
> >> example, imagine that your archive_command is an scp from the primary to 
> >> the standby. If a lightning strikes the primary before some WAL file has 
> >> been copied over to the archive directory in the standby, the standby 
> >> can't catch up. In the primary then, what's the point for a commit to 
> >> wait for transfer, if the reply from the standby doesn't guarantee that 
> >> the transaction is safe in the standby?
> > 
> > The WAL files will have already left the primary. 
> > 
> > Timeline is this in my understanding
> > 1 [Primary] Set up continuous archiving 
> > 2 [Primary] Take base backup
> > 3 [Standby] Connect to primary to initiate streaming
> > 4 [Primary] Log switch and, optionally, turn off archiving
> > 5 [Standby] Begin replaying files, initially from archive
> > 6 [Standby] Switch to replaying WAL records immediately after streaming
> > 
> > So sync rep would turn on after step 4, so that all intermediate WAL
> > files have been sent to the archive.  If we lose the Primary after this
> > point then all transactions are accessible to standby. If we lose the
> > Standby or Archive, then we need to replace them and re-run the above.
> 
> Between steps 4 and 5, there's no guarantee that all WAL files generated 
> after step 3 and the start of streaming have already been archived. 
> There's a delay between writing a WAL file and when the file has been 
> safely archived. If you lose the primary during that window, the standby 
> will have old WAL files in the archive, the most recent ones in received 
> by walreceiver, but it's missing the WAL files generated just before the 
> switch to streaming mode.

I was presuming that the synchronisation was clear, but I'm sorry it
wasn't. Sync rep would begin only *after* the last WAL file was
archived.

> > Recent changes I have requested in the architecture are:
> > * making archiving optional on primary, so we don't need to send WAL
> > data *twice*. 
> 
> Agreed. I'm not so much worried about the bandwidth, but it's a lot of 
> extra work from administration point of view. It's very hard to get it 
> right, so that you eliminate windows like the above.
> 
> As the patch stands, if you turn off archiving in the primary, and the 
> standby ever disconnects, even for only a few seconds, the standby will 
> miss any WAL generated until it reconnects, and without archiving 
> there's no way for the standby to get hold of the missed WAL.

I described earlier that archiving would turn back on again if the
replication ever failed (with correct synchronisation).

All I've asked for is the ability to turn on and turn back on archiving,
yes, with synchronisation so its safe. 

Personally, I think people will laugh if we tell them we decided to ship
all the data twice and couldn't see another way. That's the kind of
thing people give presentations at PGcon about...

> > * allowing streaming/startup process to work together via shared memory,
> > to reduce average replication delay and improve performance
> > * skip archiving/de-archiving step on standby because it's superfluous
> > (all on this thread)
> > 
> > All of those are fairly minor code changes, but reduce complexity of
> > solution and significantly reduce the amount of copying of WAL files (3
> > copy actions to/from archive removed without loss of robustness). I
> > would have made the suggestions earlier but it wasn't until I saw the
> > architecture diagrams that I understood the intention of the code.
> 
> To make archiving optional in the primary, I don't see any other choice 
> than adding the capability for the standby to request arbitrary WAL 
> files from the primary, over the wire. 

I don't think that's the only or even a desirable way. We cannot allow a
build up of WAL files to occur on the primary.

Making archiving optional isn't the big deal you're saying it is.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support