practical Fail-over methods (was: streaming replication trigger file)

On 16/06/11 18:44, John R Pierce wrote:
> On 06/16/11 1:31 AM, AI Rumman wrote:
>> When I manually create the C:\\pg\\stopreplication\\standby.txt' file,
>> then it is working. That is, B is becoming the master.
>> So, my question is, how this trigger file should be created so that B
>> will become master automatically as soon as A goes down?
>
> you need cluster management software, such as Heartbeat (on linux) or
> Veritas Cluster Service (on various operating systems) configured to
> detect system failure, and reconfigure the slave to be the master. This
> software also generally is used to manage things like the shared IP
>
> really really REALLY important is what to do when the failed original
> master is brought back up. it can NOT be allowed to wake up thinking its
> still a master since its lacking any updates since it went down, instead
> it has to be reconfigured to be a new slave.

Just following on a bit from this..

It seems fairly hard to get that part right - ensuring that the
former-master stays down, and also that automatic scripts don't run off
and convert the wrong one into a new standby either.

Do you mind if I ask how other people are doing this? Are you finding
heartbeat, keepalived or pacemaker better? What sort of ways are you
triggering the fail-over by?

And, how are you avoiding having a former-master come back up thinking
it's still the master.. yet still coping with a situation where, say,
first one machine, then the other, reboot. (thus easily triggering a
failover to one, then it going down and the other coming up and looking
like it's alone in the world)

Thanks!
Toby