Re: Issues with Quorum Commit

Markus Wanner <markus@bluegap.ch> writes:
> On 10/06/2010 04:31 AM, Simon Riggs wrote:
>> That situation would require two things
>> * First, you have set up async replication and you're not monitoring it
>> properly. Shame on you.
>
> The way I read it, Jeff is complaining about the timeout you propose
> that effectively turns sync into async replication in case of a failure.
>
> With a master that waits forever, the standby that's newly required for
> quorum certainly still needs its time to catch up. But it wouldn't live
> in danger of being "optimized away" for availability in case it cannot
> catch up within the given timeout. It's a tradeoff between availability
> and durability.

What is necessary here is a clear view on the possible states that a
standby can be in at any time, and we must stop trying to apply to
some non-ready standby the behavior we want when it's already in-sync.

From my experience operating londiste, those states would be:
1. base-backup  — self explaining2. catch-up     — getting the WAL to catch up after base backup3. wanna-sync   — don't
yethave all the WAL to get in sync4. do-sync      — all WALs are there, coming soon5. ok (async | recv | fsync | reply
—feedback loop engaged) 

So you only consider that a standby is a candidate for sync rep when
it's reached the ok state, and that's when it's able to fill the
feedback loop we've been talking about. Standby state != ok, no waiting
no nothing, it's *not* a standby as far as the master is concerned.

The other states allow to manage accepting a new standby into an
existing setup, and to manage error failures. When we stop receiving the
feedback loop events, the master knows the slave ain't in the "ok" state
any more and can demote it to "wanna-sync", because it has to keep WALs
until the slave comes again.

If the standby is not back online and wal_keep_segments makes it so that
we can't keep its wal anymore, the state gets back to "base-backup".

Not going into every details here (for example, we might need some
protocol arbitrage for the standby to be able to explain the master that
it's ok even if the master thinks it's not), but my point is that
without a clear list of standby states, we're going to hinder the master
in situations where it makes no sense to do so.

> Or do you envision any use case that requires a quorum of X standbies
> for normal operation but is just fine with only none to (X-1) standbies
> in case of failures? IMO that's when sync replication is most needed and
> when it absolutely should hold to its promises - even if it means to
> stop the system.
>
> There's no point in continuing operation if you cannot guarantee the
> minimum requirements for durability. If you happen to want such a thing,
> you should better rethink your minimum requirement (as performance for
> normal operations might benefit from a lower minimum as well).

This part of the discussion made me think of yet another refinement on
the Quorum Commit idea, even if I'm beginning to think that can be
material for later.

Basic Quorum Commit is having each transaction on the master wait for a
total number of votes to accept the transaction synced. Each standby has
a weight, meaning 1 or more votes. The problem is the flexibility isn't
there, some cases are impossible to setup. Also people want to be able
to specify their favorite standby and that's quickly uneasy.

Idea : segment the votes into "colors" or any categories you like. Have
each standby be a member of a category list, and require per-category
quorums to be reached. This is the same as attributing roles to standbys
and saying that they're all equivalent as soon as part of the given
role, with the added flexibility that you can sometime want more than
one standby of a given role to take part of the quorum.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support