Re: [PATCH 10/16] Introduce the concept that wal has a 'origin' node

On Wednesday, June 20, 2012 03:02:28 PM Robert Haas wrote:
> On Wed, Jun 20, 2012 at 5:15 AM, Andres Freund <andres@2ndquadrant.com> 
wrote:
> > One bit is fine if you have only very simple replication topologies. Once
> > you think about globally distributed databases its a bit different. You
> > describe some of that below, but just to reiterate:
> > Imagine having 6 nodes, 3 on one of two continents (ABC in north america,
> > DEF in europe). You may only want to have full intercontinental
> > interconnect between two of those (say A and D). If you only have one
> > bit to represent the origin thats not going to work because you won't be
> > able discern the changes from BC on A from the changes from those
> > originating on DEF.
> 
> I don't see the problem.  A certainly knows via which link the LCRs
> arrived.

> So: change happens on A.  A sends the change to B, C, and D.  B and C
> apply the change.  One bit is enough to keep them from regenerating
> new LCRs that get sent back to A.  So they're fine.  D also receives
> the changes (from A) and applies them, but it also does not need to
> regenerate LCRs.  Instead, it can take the LCRs that it has already
> got (from A) and send those to E and F.

> Or: change happens on B.  B sends the changes to A.  Since A knows the
> network topology, it sends the changes to C and D.  D sends them to E
> and F.  Nobody except B needs to *generate* LCRs.  All any other node
> needs to do is suppress *redundant* LCR generation.
> 
> > Another topology which is interesting is circular replications (i.e.
> > changes get shipped A->B, B->C, C->A) which is a sensible topology if
> > you only have a low change rate and a relatively high number of nodes
> > because you don't need the full combinatorial amount of connections.
> 
> I think this one is OK too.  You just generate LCRs on the origin node
> and then pass them around the ring at every step.  When the next hop
> would be the origin node then you're done.
> 
> I think you may be imagining that A generates LCRs and sends them to
> B.  B applies them, and then from the WAL just generated, it produces
> new LCRs which then get sent to C. 
Yes, thats what I am proposing.

> If you do that, then, yes,
> everything that you need to disentangle various network topologies
> must be present in WAL.  But what I'm saying is: don't do it like
> that.  Generate the LCRs just ONCE, at the origin node, and then pass
> them around the network, applying them at every node.  Then, the
> information that is needed in WAL is confined to one bit: the
> knowledge of whether or not a particular transaction is local (and
> thus LCRs should be generated) or non-local (and thus they shouldn't,
> because the origin already generated them and thus we're just handing
> them around to apply everywhere).
Sure, you can do it that way, but I don't think its a good idea. If you do it 
my way you *guarantee* that when replaying changes from node B on node C you 
have replayed changes from A at least as far as B has. Thats a really nice 
property for MM.
You *can* get same with your solution but it starts to get complicated rather 
fast. While my/our proposed solution is trivial to implement.

Andres
-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services