Timing issue in pg_basebackup
Timing issue in pg_basebackup
Hey! I recently came across this issue when running pg_basebackup from 2 nodes at the same time through ansible. One of the nodes sometimes reported: "requested WAL segment ... has already been removed". Tested on v16, v17 and v18. The workaround was running them in sequence or pause between runs. It looks like there's a timing issue here: 1. Backup A starts and runs a checkpoint - needs WAL from segment 2. 2. Backup A hasn't reserved that WAL yet. 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) - deletes segment 2. 4. Backup A asks for segment 2, which is gone, and fails. I've included a patch that moves the replication slot handing to run before the backup part as a separate function. Some possible caveats already: * Since moving the slot creation, we might run in to trouble if for example, --checkpoint=spread is used since the slot is open for longer. * If any error arises in the backup part, now there will be a lingering slot when previously no slot was created. *** DISCLAIMER *** I've used an LLM to help me write the test - I'm rubbish at Perl. Love to get some feedback. Regards, Jesper St John
Re: Timing issue in pg_basebackup
От:
Kirill Reshke <reshkekirill@gmail.com>
Дата:
On Mon, 19 Oct 2026 at 19:33, Jesper St John wrote: > > Hey! > > I recently came across this issue when running pg_basebackup from 2 > nodes at the same time through ansible. One of the nodes sometimes > reported: "requested WAL segment ... has already been removed". Tested > on v16, v17 and v18. The workaround was running them in sequence or > pause between runs. > > It looks like there's a timing issue here: > 1. Backup A starts and runs a checkpoint - needs WAL from segment 2. > 2. Backup A hasn't reserved that WAL yet. > 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) - > deletes segment 2. > 4. Backup A asks for segment 2, which is gone, and fails. Hi! Looks like this is a real issue. > I've included a patch that moves the replication slot handing to run > before the backup part as a separate function. > > Some possible caveats already: > * Since moving the slot creation, we might run in to trouble if for > example, --checkpoint=spread is used since the slot is open for longer. > * If any error arises in the backup part, now there will be a lingering > slot when previously no slot was created. lingering slots looks like a real regression here. can we remove them when basebackup client disconnects? NIT: comment in PrepareLogStreamer loses its meaning after code motion: + /* Get a second connection */ + bgconn = GetConnection(); > *** DISCLAIMER *** > I've used an LLM to help me write the test - I'm rubbish at Perl. OK -- Best regards, Kirill Reshke
Re: Timing issue in pg_basebackup
От:
Jesper St John <jesper@stjohn.se>
Дата:
On 2026-10-20 23:11, Kirill Reshke wrote: > On Mon, 19 Oct 2026 at 19:33, Jesper St John wrote: >> >> Hey! >> >> I recently came across this issue when running pg_basebackup from 2 >> nodes at the same time through ansible. One of the nodes sometimes >> reported: "requested WAL segment ... has already been removed". Tested >> on v16, v17 and v18. The workaround was running them in sequence or >> pause between runs. >> >> It looks like there's a timing issue here: >> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2. >> 2. Backup A hasn't reserved that WAL yet. >> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) >> - >> deletes segment 2. >> 4. Backup A asks for segment 2, which is gone, and fails. > > Hi! > Looks like this is a real issue. Thanks for looking into it. > >> I've included a patch that moves the replication slot handing to run >> before the backup part as a separate function. >> >> Some possible caveats already: >> * Since moving the slot creation, we might run in to trouble if for >> example, --checkpoint=spread is used since the slot is open for >> longer. >> * If any error arises in the backup part, now there will be a >> lingering >> slot when previously no slot was created. > > lingering slots looks like a real regression here. can we remove them > when basebackup client disconnects? A lingering slot could also happen today - only with this patch there's a higher risk of it happening. Would some cleanup code in a function and use it with atexit() work? > > NIT: comment in PrepareLogStreamer loses its meaning after code motion: > > + /* Get a second connection */ > + bgconn = GetConnection(); > I've moved the code to its own function - the comment were there already. > >> *** DISCLAIMER *** >> I've used an LLM to help me write the test - I'm rubbish at Perl. > > OK
Re: Timing issue in pg_basebackup
От:
Kirill Reshke <reshkekirill@gmail.com>
Дата:
On Wed, 21 Oct 2026 at 22:16, Jesper St John wrote: > > On 2026-10-20 23:11, Kirill Reshke wrote: > > > >> I've included a patch that moves the replication slot handing to run > >> before the backup part as a separate function. > >> > >> Some possible caveats already: > >> * Since moving the slot creation, we might run in to trouble if for > >> example, --checkpoint=spread is used since the slot is open for > >> longer. > >> * If any error arises in the backup part, now there will be a > >> lingering > >> slot when previously no slot was created. > > > > lingering slots looks like a real regression here. can we remove them > > when basebackup client disconnects? > > A lingering slot could also happen today - only with this patch there's > a higher risk of it happening. Higher risk of lingering slots is also a regression. But looks like this risk is anyway close to zero. > Would some cleanup code in a function and use it with atexit() work? Something like this should do, can be made a separate patch thought > > > > NIT: comment in PrepareLogStreamer loses its meaning after code motion: > > > > + /* Get a second connection */ > > + bgconn = GetConnection(); > > > > I've moved the code to its own function - the comment were there > already. > Yep, the point is It looks very out-of-context here. Like, the first line of the function says we're taking a second connection. Second to what (to basebackup)? Maybe rephrase here as 'Get a wal-serving connection' or something. -- Best regards, Kirill Reshke
Re: Timing issue in pg_basebackup
Hi! Here are 2 v2-patches as per your suggestion. I'm a bit unsure if the check in drop_lingering_replication_slot_atexit() function is enough to not break anything. Or too much. Suggestions welcome, could also make it into 1 patch if you prefer that. //Jesper On 2026-10-22 07:44, Kirill Reshke wrote: > On Wed, 21 Oct 2026 at 22:16, Jesper St John wrote: >> >> On 2026-10-20 23:11, Kirill Reshke wrote: > >> > >> >> I've included a patch that moves the replication slot handing to run >> >> before the backup part as a separate function. >> >> >> >> Some possible caveats already: >> >> * Since moving the slot creation, we might run in to trouble if for >> >> example, --checkpoint=spread is used since the slot is open for >> >> longer. >> >> * If any error arises in the backup part, now there will be a >> >> lingering >> >> slot when previously no slot was created. >> > >> > lingering slots looks like a real regression here. can we remove them >> > when basebackup client disconnects? >> >> A lingering slot could also happen today - only with this patch >> there's >> a higher risk of it happening. > > Higher risk of lingering slots is also a regression. But looks like > this risk is anyway close to zero. > >> Would some cleanup code in a function and use it with atexit() work? > > Something like this should do, can be made a separate patch thought > >> > >> > NIT: comment in PrepareLogStreamer loses its meaning after code motion: >> > >> > + /* Get a second connection */ >> > + bgconn = GetConnection(); >> > >> >> I've moved the code to its own function - the comment were there >> already. >> > > Yep, the point is It looks very out-of-context here. Like, the first > line of the function says we're taking a second connection. Second to > what (to basebackup)? Maybe rephrase here as 'Get a wal-serving > connection' or something.
Re: Timing issue in pg_basebackup
Added this patch to the CF app. Made it into one and rebased it. //Jesper On 2026-10-22 07:44, Kirill Reshke wrote: > On Wed, 21 Oct 2026 at 22:16, Jesper St John wrote: >> >> On 2026-10-20 23:11, Kirill Reshke wrote: > >> > >> >> I've included a patch that moves the replication slot handing to run >> >> before the backup part as a separate function. >> >> >> >> Some possible caveats already: >> >> * Since moving the slot creation, we might run in to trouble if for >> >> example, --checkpoint=spread is used since the slot is open for >> >> longer. >> >> * If any error arises in the backup part, now there will be a >> >> lingering >> >> slot when previously no slot was created. >> > >> > lingering slots looks like a real regression here. can we remove them >> > when basebackup client disconnects? >> >> A lingering slot could also happen today - only with this patch >> there's >> a higher risk of it happening. > > Higher risk of lingering slots is also a regression. But looks like > this risk is anyway close to zero. > >> Would some cleanup code in a function and use it with atexit() work? > > Something like this should do, can be made a separate patch thought > >> > >> > NIT: comment in PrepareLogStreamer loses its meaning after code motion: >> > >> > + /* Get a second connection */ >> > + bgconn = GetConnection(); >> > >> >> I've moved the code to its own function - the comment were there >> already. >> > > Yep, the point is It looks very out-of-context here. Like, the first > line of the function says we're taking a second connection. Second to > what (to basebackup)? Maybe rephrase here as 'Get a wal-serving > connection' or something.
Re: Timing issue in pg_basebackup
От:
Joao Rito <joaovictor.rito@gmail.com>
Дата:
Hi Jesper, Thank you for the patch. I tested v3-0001 (CF entry 7613) and here are my results. Setup: master at c1694b7 (11 Nov 2026), configured with --enable-cassert, --enable-tap-tests and --enable-injection-points, on Linux (WSL Ubuntu). The patch applied cleanly with git apply. 1. The bug on unpatched master I held the walsender at the existing injection point replication-slot-create-begin (so BASE_BACKUP has returned, but the slot does not exist yet). Then I switched WAL four times, ran CHECKPOINT, and released the walsender. The server was running with wal_keep_size=0. pg_basebackup: error: unexpected termination of replication stream: ERROR: requested WAL segment 000000010000000000000002 has already been removed pg_basebackup: error: background process terminated unexpectedly pg_basebackup exited with 1. So the race you describe is real, and it can be triggered on demand. 2. With v3 applied The same script gives pg_basebackup exit 0, no error in stderr or in the server log, "backup successfully verified" from pg_verifybackup, and no leftover slot. 3. Tests "make -C src/bin/pg_basebackup check" with the patch: all 7 files pass (372 tests), including the new 013_wal_slot_timing.pl and the new cases in 010_pg_basebackup.pl. I ran 013 alone with prove -v to confirm it was not skipped: "ok 1 - pg_basebackup succeeds with WAL activity during slot creation", "ok 2 - no WAL needed by the backup was removed", "ok 3 - backup with streamed WAL verifies", Files=1, Tests=3, Result: PASS. As a negative control, I kept the new tests but put back the original pg_basebackup.c. Then 013 fails all 3 subtests with the same "has already been removed" error, so the test does detect the bug. 4. A note on lingering slots and signals You and Kirill discussed the lingering slot in the thread. I tried to see what happens on a signal. With v3 the slot is created before the checkpoint, so the window in which an interrupted pg_basebackup can leave the slot behind now also covers the checkpoint wait. With --checkpoint=spread that can be long. To compare fairly, I did the test after the slot exists on both builds (on master the slot is only created after BASE_BACKUP returns, so a SIGTERM during the checkpoint finds no slot there). I ran pg_basebackup -Xstream --create-slot -S s1 --checkpoint=fast --max-rate=1M on a 216 MB database, polled pg_replication_slots until s1 appeared, and then sent SIGTERM while the backup was still running: - master: s1 was still there after SIGTERM (and still there 11 s later). - with v3: s1 was still there after SIGTERM (and still there 11 s later). So master also leaves the slot behind on SIGTERM once the slot exists. I think this is expected, since an atexit() handler probably does not run when the process is killed by a signal, but I did not test that part directly. v3 does not make this worse for a signal after the slot is created. It only starts the window earlier, so it is longer in time (the checkpoint wait). The slot is not temporary here because I used --create-slot. The walsender state differed between runs: in the first run s1 showed as inactive right after the kill with v3 (active on master), and in a second run s1 was still active on both builds 11 s after the kill. Your atexit() handler works for the error cases covered by the TAP tests. If you want to cover signals too, maybe a signal handler could drop the slot, or the docs for --create-slot / --checkpoint could say that a slot may remain if the process is killed. This is just a suggestion, I do not see it as a regression of the patch. I only tested SIGTERM. Small comment on the code: the check "success || in_log_streamer || bgchild > 0" looks fine to me in the cases I ran, but I did not review all exit paths in detail. What I did NOT test: - Other platforms (Windows, where pg_basebackup uses a thread instead of fork, and the new atexit() and bgconn handling matter). - Other WAL methods and formats (--wal-method=fetch/none, tar format, compressed WAL, --format=tar with stream) beyond what the existing TAP suite covers. - Older server versions (the MINIMUM_VERSION_FOR_TEMP_SLOTS branch). - SIGINT, and failures after the background WAL receiver has started. - Meson build (I used make/autoconf only), and the docs/pgindent formatting. - The real two-node ansible scenario from your first mail (I only reproduced the race with an injection point), and the performance effect of holding the slot during long spread checkpoints (only the slot visibility was checked, above). - Whether other tools that use the same pattern (for example pg_receivewal --create-slot) need the same change. The scripts and the full log are available if you want them. Thanks again for working on this. Best regards, Joao Rito
Re: Timing issue in pg_basebackup
От:
Zsolt Parragi <zsolt.parragi@percona.com>
Дата:
Hello I can also confirm the race on master and that v3 fixes it. However: > * Since moving the slot creation, we might run in to trouble if for > example, --checkpoint=spread is used since the slot is open for longer. Moving the slot creation in front of BASE_BACKUP causes other new failures, exactly like the concern you mentioned in the first email, and I don't think overall it is better than the current issue. 1. The new slot can be invalidated by the backup's own checkpoint A physical slot starts with the redo pointer of the previous checkpoint. With the patch it is created, released (so it is inactive), and stays like that for the whole start-of-backup checkpoint. At the end of that checkpoint we call InvalidateObsoleteReplicationSlots(), and our slot is one of the candidates for invalidation, and can be invalidated based on max_slot_wal_keep_size or idle_replication_slot_timeout. This results in backup failures in multiple scenarios, with server log entries such as DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes. 2. The second connection is killed by idle_session_timeout bgconn is now opened before BASE_BACKUP and sits idle in walsender command mode until the checkpoint is done. PostgresMain() arms IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide idle_session_timeout set and a checkpoint longer than that, the walsender exits (dropping the temporary slot, and with it the protection the patch adds), and the backup fails after the checkpoint: pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM": FATAL: terminating connection due to idle-session timeout (2) is fixable, but (1) is a more tricky problem and I trickier to solve. Maybe immediately after the slot is created we could use READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn? Alternatively, instead of reordering, since this is an unlikely race, maybe early detection and retry could be also an option instead of reordering the operations? After creating the slot, check restart_lsn: if it is behond startptr's segment then drop the connection and restart.
Re: Timing issue in pg_basebackup
От:
Magnus Hagander <magnus@hagander.net>
Дата:
On Thu, 12 Nov 2026 at 22:19, Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Hello
I can also confirm the race on master and that v3 fixes it. However:
> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.
Moving the slot creation in front of BASE_BACKUP causes other new
failures, exactly like the concern you mentioned in the first email,
and I don't think overall it is better than the current issue.
1. The new slot can be invalidated by the backup's own checkpoint
A physical slot starts with the redo pointer of the previous
checkpoint. With the patch it is created, released (so it is
inactive), and stays like that for the whole start-of-backup
checkpoint. At the end of that checkpoint we call
InvalidateObsoleteReplicationSlots(), and our slot is one of the
candidates for invalidation, and can be invalidated based on
max_slot_wal_keep_size or idle_replication_slot_timeout.
This results in backup failures in multiple scenarios, with server log
entries such as
DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes.
2. The second connection is killed by idle_session_timeout
bgconn is now opened before BASE_BACKUP and sits idle in walsender
command mode until the checkpoint is done. PostgresMain() arms
IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
idle_session_timeout set and a checkpoint longer than that, the
walsender exits (dropping the temporary slot, and with it the
protection the patch adds), and the backup fails after the checkpoint:
pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
FATAL: terminating connection due to idle-session timeout
(2) is fixable, but (1) is a more tricky problem and I trickier to
solve. Maybe immediately after the slot is created we could use
READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?
Alternatively, instead of reordering, since this is an unlikely race,
maybe early detection and retry could be also an option instead of
reordering the operations? After creating the slot, check restart_lsn:
if it is behond startptr's segment then drop the connection and
restart.
Per some off-list discussion with Jesper. I don't think that wil work, as READ_REPLICATION_SLOT won't actually advance the last use time on it, so it would still time out.
I think the correct thing to do here is to start the full log streamer *before* starting the base backup. This will lead to including just a little bit unnecessarily much WAL in the backup, but it won't be much - we already need to keep the WAL back to the start of the base backup checkpoint start. So the only extra WAL to be included would be between the log streaming started and the base backup command running, which would not be much. The only real difference would be the streaming starts already during the checkpoint, so with a slow cheakpoing it won't start the streamer with a large backlog.
Re: Timing issue in pg_basebackup
От:
Andrey Borodin <x4mmm@yandex-team.ru>
Дата:
Hi Magnus, all, Nick reported the same startpoint/slot race in August [0], tracked in CF 7233 [1]. The client patch there opens the streaming connection and creates its slot before BASE_BACKUP, but starts streaming only after receiving the backup startpoint. Your proposal goes further than that. We also discussed cleanup of a newly created permanent slot when BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and idle timeout are relevant to that approach too. Could we combine the work and keep both threads linked from one CF entry? This thread is currently tracked separately as CF 7613 [2]. Thank you! Best regards, Andrey Borodin. [0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com [1] https://commitfest.postgresql.org/patch/7233/ [2] https://commitfest.postgresql.org/patch/7613/
Re: Timing issue in pg_basebackup
От:
Jesper St John <admin@zonezero.se>
Дата:
Hi! Sorry, even though I’m pretty sure I searched for similar issues before posting, I missed it. Big thanks to Zsolt, Joao and Kirill for your input. I think combining work sounds like the reasonable thing to do. I’d be happy to help out in either thread. If any of you already know if one patch is preferable over the other, please do tell. If not I’ll review and try posting a follow-up. I’ve slowly started to implement a “start streaming directly after slot creation”-functionality and perhaps that could work with either patch. //Jesper Sent from my iPhone > On 16 Nov 2026, at 13:09, Andrey Borodin wrote: > Hi Magnus, all, > > Nick reported the same startpoint/slot race in August [0], tracked in > CF 7233 [1]. The client patch there opens the streaming connection and > creates its slot before BASE_BACKUP, but starts streaming only after > receiving the backup startpoint. Your proposal goes further than that. > > We also discussed cleanup of a newly created permanent slot when > BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and > idle timeout are relevant to that approach too. > > Could we combine the work and keep both threads linked from one CF > entry? This thread is currently tracked separately as CF 7613 [2]. > > Thank you! > > Best regards, Andrey Borodin. > > [0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com > [1] https://commitfest.postgresql.org/patch/7233/ > [2] https://commitfest.postgresql.org/patch/7613/