Hi!
In a cluster set up with postgres 9.6, streaming replication and
repmgr I'm struggling to find a good/simple solution for avoiding
split brain.
The current theoretical setup consists of 4 nodes across two data
centers. The master node is setup with 1 of 3 synchronous replication.
That is it waits for at least one other node to COMMIT as well.
repmgrd is installed on every node.
The clients will use postgresql JDBC with targetServerType=master so
they connect only to the master server in a list of four hosts.
The split brain scenario I forsee is when the master node locks up or
is isolated for a while and comes back online after repmgrd on other
nodes have elected a new master.
As the original master node has a requirement of one synced
replication node and the remaining two standbys are streaming from the
new master it will fortunately not start writing a separate timeline,
but will still serve dated read only queries. For writes it will
accept connections which hang. The repmgrd instance on the original
master sees no problem either so does nothing.
Ideally though this instance should be shut down as it has no slaves
attached and the status on other nodes indicates this master node is
failed.
Any suggestions? I'm trying to keep the setup simple without a central
pgbouncer/pgpool. Any simple way to avoid a central connection point
or custom monitoring script that looks for exactly this issue?
Also, do you see any other potential pitfalls in this setup?
Thanks for thinking this through,
Aleksander
--
Aleksander Kamenik