Re: Issues with Quorum Commit

On Tue, 2010-10-05 at 22:32 +0300, Heikki Linnakangas wrote:
> On 05.10.2010 22:11, Josh Berkus wrote:
> > There's been a lot of discussion on synch rep lately which involves
> > quorum commit.  I need to raise some major design issues with quorum
> > commit which I don't think that people have really considered, and may
> > be sufficient to prevent it from being included in 9.1.
> 
> Thanks for bringing these up.

Yes, I'm very happy to discuss these.

The points appear to be directed at "quorum commit", which is a name
I've used. But most of the points apply more to Fujii's patch than my
own. I can only presume that Josh wants to prevent us from adopting a
design that allows sync against multiple standbys.

> > A. Permanent Synchronization Failure
> > ---------------------------------
> > Quorum commit, like other forms of more-than-one-standby synch rep,
> > offers the possibility that one or more standbys could end up
> > irretrievably desyncronized with the master.
> >
> > 1. Quorum is 3 servers (out of 5) with mode "apply"
> > 2. Standbys 2 and 4 receive and apply transaction # 20001.
> > 3. Due to a network issue, no other standby applies #20001.
> > 4. Accordingly, the master rolls back #20001 and cancels, either due to
> > timeout or DBA cancel.
> 
> The master can not roll back or cancel the transaction. That's 
> completely infeasible, the WAL record has been written to local disk 
> already. The best it can do is halt and wait for enough standbys to 
> appear to fulfill the quorum. The client will hang waiting for the 
> COMMIT to finish, and the transaction will appear as in-progress to 
> other transactions.

Yes, that point has long been understood. Neither patch does this, and
in fact the issue is a completely general one.

> There's subtle point here that I don't think has been discussed yet: If 
> the master is forcibly restarted at that point, with pg_ctl restart -m 
> immediate, strictly speaking the master should start up in the same 
> state, with the unlucky transaction still appearing as in-progress, 
> until the standby acknowledges.

That is a very important point, but again, nothing to do with quorum
commit. For strict correctness, we should do that. Are you suggesting we
should do that here? 

> > 5. #2 and #5 are now hopelessly out of synch with the master.
> 
> > B. Eventual Inconsistency
> > -------------------------
> > If we have a quorum commit, it's possible for any individual standby to
> > be indefinitely ahead of any standby which is not needed by the quorum.
> >   This means that:
> >
> > -- There is no clear criteria for when a standby which is not needed for
> > quorum should be considered no longer a synch standby, and
> > -- Applications cannot make assumptions that synch rep promises some
> > specific window of synchronicity, eliminating a lot of the value of
> > quorum commit.
> 
> Yep.

Could the person that wrote that actually explain what a "specific
window of synchronicity" is? I'm not sure whether to agree, or disagree.

> > C. Performance
> > --------------
> > Doing quorum commit requires significant extra accounting on the
> > master's part: it must keep track of how many standbys committed for
> > each pending transaction (and remember there may be many at the same
> > time).
> >
> > Doing so could involve significant response-time overhead added to the
> > simple case where there is only one standby, as well as memory usage,
> > and likely a lot of troubleshooting of the mechanism from us.
> 
> My gut feeling is that overhead will pale to insignificance compared to 
> the network and other overheads of actually getting the WAL to the 
> standby and processing the acknowledgments.

You're ignoring Josh's points. Those exact points have been made by me
in support of the design of my patch and against Fujii's. The mechanism
to do this will be more complex and more likely to break. And it will be
slower and that is a concern for me.

> > D. Adding/Replacing Quorum Members
> > ----------------------------------
> > For Quorum commit to be really valuable, we need to be able to add new
> > quorum members and remove dead ones *without stopping the master*.  Per
> > discussion about the startup issues with only one master, we have not
> > worked out how to do this for synch rep standbys.  It's reasonable to
> > assume that this will be more complex for a quorum group than with a
> > single synch standby.
> >
> > Consider the case, for example, where due to a network outage we have
> > dropped below quorum.  What is the strategy for getting the system
> > running again by adding standbys?
> 
> You start a new one from the latest base backup and let it catch up? 
> Possibly modifying the config file in the master to let it know about 
> the new standby, if we go down that path. This part doesn't seem 
> particularly hard to me.

Agreed, not sure of the issue there.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Development, 24x7 Support, Training and Services