On Thu, 2024-03-28 at 15:52 +0100, Emond Papegaaij wrote:
> * we detach the primary database backend, forcing a failover
> * pgpool selects a new primary database and promotes it
> * the other 2 nodes (the old primary and the other standby) are rewound
> and streaming is resumed from the new primary
> * the node that needed to be taken out of the cluster (the old primary)
> is shutdown and rebooted
>
> This works fine most of the time, but sometimes we see this message on one of the nodes:
> pg_rewind: source and target cluster are on the same timeline pg_rewind: no rewind required
> This message seems timing related, as the first node might report that,
> while the second reports something like:
> pg_rewind: servers diverged at WAL location 5/F28AB1A8 on timeline 21
> pg_rewind: rewinding from last common checkpoint at 5/F27FCA98 on timeline 21
> pg_rewind: Done!
>
> If we ignore the response from pg_rewind, streaming will break on the node that reported
> no rewind was required. On the new primary, we do observe the database moving from timeline
> 21 to 22, but it seems this takes some time to materialize to be observable by pg_rewind.
>
> 1. Is my observation about the starting of a new timeline correct?
> 2. If yes, is there anything we can do during to block promotion process until the new
> timeline has fully materialized, either by waiting or preferably forcing the new
> timeline to be started?
This must be the problem addressed by commit 009eeee746 [1].
You'd have to upgrade to PostgreSQL v16, which would be a good idea anyway, given
that you are running v12.
A temporary workaround could be to explicitly trigger a checkpoint right after
promotion.
Yours,
Laurenz Albe
[1]. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=009eeee746825090ec7194321a3db4b298d6571e