Timing issue in pg_basebackup

Поиск
Список
Период
Сортировка
Искать

Timing issue in pg_basebackup

От:
Jesper St John <jesper@stjohn.se>
Дата:
Вложения:
Hey!

I recently came across this issue when running pg_basebackup from 2 
nodes at the same time through ansible. One of the nodes sometimes 
reported: "requested WAL segment ... has already been removed". Tested 
on v16, v17 and v18. The workaround was running them in sequence or 
pause between runs.

It looks like there's a timing issue here:
1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
2. Backup A hasn't reserved that WAL yet.
3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) - 
deletes segment 2.
4. Backup A asks for segment 2, which is gone, and fails.

I've included a patch that moves the replication slot handing to run 
before the backup part as a separate function.

Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for 
example, --checkpoint=spread is used since the slot is open for longer.
* If any error arises in the backup part, now there will be a lingering 
slot when previously no slot was created.

*** DISCLAIMER ***
I've used an LLM to help me write the test - I'm rubbish at Perl.

Love to get some feedback.

Regards,
Jesper St John

Re: Timing issue in pg_basebackup

От:
Kirill Reshke <reshkekirill@gmail.com>
Дата:
On Mon, 19 Oct 2026 at 19:33, Jesper St John  wrote:
>
> Hey!
>
> I recently came across this issue when running pg_basebackup from 2
> nodes at the same time through ansible. One of the nodes sometimes
> reported: "requested WAL segment ... has already been removed". Tested
> on v16, v17 and v18. The workaround was running them in sequence or
> pause between runs.
>
> It looks like there's a timing issue here:
> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
> 2. Backup A hasn't reserved that WAL yet.
> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) -
> deletes segment 2.
> 4. Backup A asks for segment 2, which is gone, and fails.

Hi!
Looks like this is a real issue.

> I've included a patch that moves the replication slot handing to run
> before the backup part as a separate function.
>
> Some possible caveats already:
> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.
> * If any error arises in the backup part, now there will be a lingering
> slot when previously no slot was created.

lingering slots looks like a real regression here. can we remove them
when basebackup client disconnects?

NIT: comment in PrepareLogStreamer loses its meaning after code motion:

+ /* Get a second connection */
+ bgconn = GetConnection();


> *** DISCLAIMER ***
> I've used an LLM to help me write the test - I'm rubbish at Perl.

OK



-- 
Best regards,
Kirill Reshke


Re: Timing issue in pg_basebackup

От:
Jesper St John <jesper@stjohn.se>
Дата:
On 2026-10-20 23:11, Kirill Reshke wrote:
> On Mon, 19 Oct 2026 at 19:33, Jesper St John  wrote:
>> 
>> Hey!
>> 
>> I recently came across this issue when running pg_basebackup from 2
>> nodes at the same time through ansible. One of the nodes sometimes
>> reported: "requested WAL segment ... has already been removed". Tested
>> on v16, v17 and v18. The workaround was running them in sequence or
>> pause between runs.
>> 
>> It looks like there's a timing issue here:
>> 1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
>> 2. Backup A hasn't reserved that WAL yet.
>> 3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) 
>> -
>> deletes segment 2.
>> 4. Backup A asks for segment 2, which is gone, and fails.
> 
> Hi!
> Looks like this is a real issue.

Thanks for looking into it.

> 
>> I've included a patch that moves the replication slot handing to run
>> before the backup part as a separate function.
>> 
>> Some possible caveats already:
>> * Since moving the slot creation, we might run in to trouble if for
>> example, --checkpoint=spread is used since the slot is open for 
>> longer.
>> * If any error arises in the backup part, now there will be a 
>> lingering
>> slot when previously no slot was created.
> 
> lingering slots looks like a real regression here. can we remove them
> when basebackup client disconnects?

A lingering slot could also happen today - only with this patch there's 
a higher risk of it happening.

Would some cleanup code in a function and use it with atexit() work?

> 
> NIT: comment in PrepareLogStreamer loses its meaning after code motion:
> 
> + /* Get a second connection */
> + bgconn = GetConnection();
> 

I've moved the code to its own function - the comment were there 
already.


> 
>> *** DISCLAIMER ***
>> I've used an LLM to help me write the test - I'm rubbish at Perl.
> 
> OK


Re: Timing issue in pg_basebackup

От:
Kirill Reshke <reshkekirill@gmail.com>
Дата:
On Wed, 21 Oct 2026 at 22:16, Jesper St John  wrote:
>
> On 2026-10-20 23:11, Kirill Reshke wrote:

> >
> >> I've included a patch that moves the replication slot handing to run
> >> before the backup part as a separate function.
> >>
> >> Some possible caveats already:
> >> * Since moving the slot creation, we might run in to trouble if for
> >> example, --checkpoint=spread is used since the slot is open for
> >> longer.
> >> * If any error arises in the backup part, now there will be a
> >> lingering
> >> slot when previously no slot was created.
> >
> > lingering slots looks like a real regression here. can we remove them
> > when basebackup client disconnects?
>
> A lingering slot could also happen today - only with this patch there's
> a higher risk of it happening.

Higher risk of lingering slots is also a regression. But looks like
this risk is anyway close to zero.

> Would some cleanup code in a function and use it with atexit() work?

Something like this should do, can be made a separate patch thought

> >
> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
> >
> > + /* Get a second connection */
> > + bgconn = GetConnection();
> >
>
> I've moved the code to its own function - the comment were there
> already.
>

Yep, the point is It looks very out-of-context here. Like, the first
line of the function says we're taking a second connection. Second to
what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
connection' or something.



-- 
Best regards,
Kirill Reshke


Re: Timing issue in pg_basebackup

От:
Jesper St John <jesper@stjohn.se>
Дата:
Вложения:
Hi!

Here are 2 v2-patches as per your suggestion.

I'm a bit unsure if the check in 
drop_lingering_replication_slot_atexit() function is enough to not break 
anything.
Or too much.

Suggestions welcome, could also make it into 1 patch if you prefer that.

//Jesper

On 2026-10-22 07:44, Kirill Reshke wrote:
> On Wed, 21 Oct 2026 at 22:16, Jesper St John  wrote:
>> 
>> On 2026-10-20 23:11, Kirill Reshke wrote:
> 
>> >
>> >> I've included a patch that moves the replication slot handing to run
>> >> before the backup part as a separate function.
>> >>
>> >> Some possible caveats already:
>> >> * Since moving the slot creation, we might run in to trouble if for
>> >> example, --checkpoint=spread is used since the slot is open for
>> >> longer.
>> >> * If any error arises in the backup part, now there will be a
>> >> lingering
>> >> slot when previously no slot was created.
>> >
>> > lingering slots looks like a real regression here. can we remove them
>> > when basebackup client disconnects?
>> 
>> A lingering slot could also happen today - only with this patch 
>> there's
>> a higher risk of it happening.
> 
> Higher risk of lingering slots is also a regression. But looks like
> this risk is anyway close to zero.
> 
>> Would some cleanup code in a function and use it with atexit() work?
> 
> Something like this should do, can be made a separate patch thought
> 
>> >
>> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
>> >
>> > + /* Get a second connection */
>> > + bgconn = GetConnection();
>> >
>> 
>> I've moved the code to its own function - the comment were there
>> already.
>> 
> 
> Yep, the point is It looks very out-of-context here. Like, the first
> line of the function says we're taking a second connection. Second to
> what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
> connection' or something.

Re: Timing issue in pg_basebackup

От:
Jesper St John <jesper@stjohn.se>
Дата:
Вложения:
Added this patch to the CF app.

Made it into one and rebased it.

//Jesper

On 2026-10-22 07:44, Kirill Reshke wrote:
> On Wed, 21 Oct 2026 at 22:16, Jesper St John  wrote:
>> 
>> On 2026-10-20 23:11, Kirill Reshke wrote:
> 
>> >
>> >> I've included a patch that moves the replication slot handing to run
>> >> before the backup part as a separate function.
>> >>
>> >> Some possible caveats already:
>> >> * Since moving the slot creation, we might run in to trouble if for
>> >> example, --checkpoint=spread is used since the slot is open for
>> >> longer.
>> >> * If any error arises in the backup part, now there will be a
>> >> lingering
>> >> slot when previously no slot was created.
>> >
>> > lingering slots looks like a real regression here. can we remove them
>> > when basebackup client disconnects?
>> 
>> A lingering slot could also happen today - only with this patch 
>> there's
>> a higher risk of it happening.
> 
> Higher risk of lingering slots is also a regression. But looks like
> this risk is anyway close to zero.
> 
>> Would some cleanup code in a function and use it with atexit() work?
> 
> Something like this should do, can be made a separate patch thought
> 
>> >
>> > NIT: comment in PrepareLogStreamer loses its meaning after code motion:
>> >
>> > + /* Get a second connection */
>> > + bgconn = GetConnection();
>> >
>> 
>> I've moved the code to its own function - the comment were there
>> already.
>> 
> 
> Yep, the point is It looks very out-of-context here. Like, the first
> line of the function says we're taking a second connection. Second to
> what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
> connection' or something.

Re: Timing issue in pg_basebackup

От:
Joao Rito <joaovictor.rito@gmail.com>
Дата:
Hi Jesper,

Thank you for the patch. I tested v3-0001 (CF entry 7613) and here are my
results.

Setup: master at c1694b7 (11 Nov 2026), configured with --enable-cassert,
--enable-tap-tests and --enable-injection-points, on Linux (WSL Ubuntu). The
patch applied cleanly with git apply.

1. The bug on unpatched master

I held the walsender at the existing injection point
replication-slot-create-begin (so BASE_BACKUP has returned, but the slot
does not exist yet). Then I switched WAL four times, ran CHECKPOINT, and
released the walsender. The server was running with wal_keep_size=0.

  pg_basebackup: error: unexpected termination of replication stream: ERROR:
  requested WAL segment 000000010000000000000002 has already been removed
  pg_basebackup: error: background process terminated unexpectedly

pg_basebackup exited with 1. So the race you describe is real, and it can be
triggered on demand.

2. With v3 applied

The same script gives pg_basebackup exit 0, no error in stderr or in the
server log, "backup successfully verified" from pg_verifybackup, and no
leftover slot.

3. Tests

"make -C src/bin/pg_basebackup check" with the patch: all 7 files pass (372
tests), including the new 013_wal_slot_timing.pl and the new cases in
010_pg_basebackup.pl. I ran 013 alone with prove -v to confirm it was not
skipped: "ok 1 - pg_basebackup succeeds with WAL activity during slot
creation", "ok 2 - no WAL needed by the backup was removed", "ok 3 - backup
with streamed WAL verifies", Files=1, Tests=3, Result: PASS. As a negative
control, I kept the new tests but put back the original pg_basebackup.c.
Then 013 fails all 3 subtests with the same "has already been removed"
error, so the test does detect the bug.

4. A note on lingering slots and signals

You and Kirill discussed the lingering slot in the thread. I tried to see
what happens on a signal. With v3 the slot is created before the checkpoint,
so the window in which an interrupted pg_basebackup can leave the slot
behind now also covers the checkpoint wait. With --checkpoint=spread that
can be long.

To compare fairly, I did the test after the slot exists on both builds (on
master the slot is only created after BASE_BACKUP returns, so a SIGTERM
during the checkpoint finds no slot there). I ran pg_basebackup -Xstream
--create-slot -S s1 --checkpoint=fast --max-rate=1M on a 216 MB database,
polled pg_replication_slots until s1 appeared, and then sent SIGTERM while
the backup was still running:

- master: s1 was still there after SIGTERM (and still there 11 s later).
- with v3: s1 was still there after SIGTERM (and still there 11 s later).

So master also leaves the slot behind on SIGTERM once the slot exists. I
think this is expected, since an atexit() handler probably does not run when
the process is killed by a signal, but I did not test that part directly. v3
does not make this worse for a signal after the slot is created. It only
starts the window earlier, so it is longer in time (the checkpoint wait).
The slot is not temporary here because I used --create-slot. The walsender
state differed between runs: in the first run s1 showed as inactive right
after the kill with v3 (active on master), and in a second run s1 was still
active on both builds 11 s after the kill.

Your atexit() handler works for the error cases covered by the TAP tests. If
you want to cover signals too, maybe a signal handler could drop the slot,
or the docs for --create-slot / --checkpoint could say that a slot may
remain if the process is killed. This is just a suggestion, I do not see it
as a regression of the patch. I only tested SIGTERM.

Small comment on the code: the check "success || in_log_streamer || bgchild
> 0" looks fine to me in the cases I ran, but I did not review all exit
paths in detail.

What I did NOT test:
- Other platforms (Windows, where pg_basebackup uses a thread instead of
  fork, and the new atexit() and bgconn handling matter).
- Other WAL methods and formats (--wal-method=fetch/none, tar format,
  compressed WAL, --format=tar with stream) beyond what the existing TAP
  suite covers.
- Older server versions (the MINIMUM_VERSION_FOR_TEMP_SLOTS branch).
- SIGINT, and failures after the background WAL receiver has started.
- Meson build (I used make/autoconf only), and the docs/pgindent formatting.
- The real two-node ansible scenario from your first mail (I only reproduced
  the race with an injection point), and the performance effect of holding
  the slot during long spread checkpoints (only the slot visibility was
  checked, above).
- Whether other tools that use the same pattern (for example pg_receivewal
  --create-slot) need the same change.

The scripts and the full log are available if you want them. Thanks again
for working on this.

Best regards,
Joao Rito


Re: Timing issue in pg_basebackup

От:
Zsolt Parragi <zsolt.parragi@percona.com>
Дата:
Hello

I can also confirm the race on master and that v3 fixes it. However:

> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.

Moving the slot creation in front of BASE_BACKUP causes other new
failures, exactly like the concern you mentioned in the first email,
and I don't think overall it is better than the current issue.

1. The new slot can be invalidated by the backup's own checkpoint

A physical slot starts with the redo pointer of the previous
checkpoint. With the patch it is created, released (so it is
inactive), and stays like that for the whole start-of-backup
checkpoint. At the end of that checkpoint we call
InvalidateObsoleteReplicationSlots(), and our slot is one of the
candidates for invalidation, and can be invalidated based on
max_slot_wal_keep_size or idle_replication_slot_timeout.

This results in backup failures in multiple scenarios, with server log
entries such as

DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes.

2. The second connection is killed by idle_session_timeout

bgconn is now opened before BASE_BACKUP and sits idle in walsender
command mode until the checkpoint is done. PostgresMain() arms
IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
idle_session_timeout set and a checkpoint longer than that, the
walsender exits (dropping the temporary slot, and with it the
protection the patch adds), and the backup fails after the checkpoint:

pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
FATAL: terminating connection due to idle-session timeout

(2) is fixable, but (1) is a more tricky problem and I trickier to
solve. Maybe immediately after the slot is created we could use
READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?

Alternatively, instead of reordering, since this is an unlikely race,
maybe early detection and retry could be also an option instead of
reordering the operations? After creating the slot, check restart_lsn:
if it is behond startptr's segment then drop the connection and
restart.


Re: Timing issue in pg_basebackup

От:
Magnus Hagander <magnus@hagander.net>
Дата:


On Thu, 12 Nov 2026 at 22:19, Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Hello

I can also confirm the race on master and that v3 fixes it. However:
> * Since moving the slot creation, we might run in to trouble if for
> example, --checkpoint=spread is used since the slot is open for longer.

Moving the slot creation in front of BASE_BACKUP causes other new
failures, exactly like the concern you mentioned in the first email,
and I don't think overall it is better than the current issue.

1. The new slot can be invalidated by the backup's own checkpoint

A physical slot starts with the redo pointer of the previous
checkpoint. With the patch it is created, released (so it is
inactive), and stays like that for the whole start-of-backup
checkpoint. At the end of that checkpoint we call
InvalidateObsoleteReplicationSlots(), and our slot is one of the
candidates for invalidation, and can be invalidated based on
max_slot_wal_keep_size or idle_replication_slot_timeout.

This results in backup failures in multiple scenarios, with server log
entries such as

DETAIL: The slot's restart_lsn 0/8B317610 exceeds the limit by 164530672 bytes.

2. The second connection is killed by idle_session_timeout

bgconn is now opened before BASE_BACKUP and sits idle in walsender
command mode until the checkpoint is done. PostgresMain() arms
IDLE_SESSION_TIMEOUT for walsenders too, so with a cluster-wide
idle_session_timeout set and a checkpoint longer than that, the
walsender exits (dropping the temporary slot, and with it the
protection the patch adds), and the backup fails after the checkpoint:

pg_basebackup: error: could not send replication command "IDENTIFY_SYSTEM":
FATAL: terminating connection due to idle-session timeout

(2) is fixable, but (1) is a more tricky problem and I trickier to
solve. Maybe immediately after the slot is created we could use
READ_REPLICATION_SLOT and the START_REPLICATION with the restart_lsn?

Alternatively, instead of reordering, since this is an unlikely race,
maybe early detection and retry could be also an option instead of
reordering the operations? After creating the slot, check restart_lsn:
if it is behond startptr's segment then drop the connection and
restart.

Per some off-list discussion with Jesper. I don't think that wil work, as READ_REPLICATION_SLOT won't actually advance the last use time on it, so it would still time out.

I think the correct thing to do here is to start the full log streamer *before* starting the base backup. This will lead to including just a little bit unnecessarily much WAL in the backup, but it won't be much - we already need to keep the WAL back to the start of the base backup checkpoint start. So the only extra WAL to be included would be between the log streaming started and the base backup command running, which would not be much. The only real difference would be the streaming starts already during the checkpoint, so with a slow cheakpoing it won't start the streamer with a large backlog.
 
--

Re: Timing issue in pg_basebackup

От:
Andrey Borodin <x4mmm@yandex-team.ru>
Дата:
Hi Magnus, all,

Nick reported the same startpoint/slot race in August [0], tracked in
CF 7233 [1]. The client patch there opens the streaming connection and
creates its slot before BASE_BACKUP, but starts streaming only after
receiving the backup startpoint. Your proposal goes further than that.

We also discussed cleanup of a newly created permanent slot when
BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and
idle timeout are relevant to that approach too.

Could we combine the work and keep both threads linked from one CF
entry? This thread is currently tracked separately as CF 7613 [2].

Thank you!

Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com
[1] https://commitfest.postgresql.org/patch/7233/
[2] https://commitfest.postgresql.org/patch/7613/


Re: Timing issue in pg_basebackup

От:
Jesper St John <admin@zonezero.se>
Дата:
Hi!

Sorry, even though I’m pretty sure I searched for similar issues before posting, I missed it.

Big thanks to Zsolt, Joao and Kirill for your input.

I think combining work sounds like the reasonable thing to do. I’d be happy to help out in either thread.

If any of you already know if one patch is preferable over the other, please do tell. If not I’ll review and try posting a follow-up.

I’ve slowly started to implement a “start streaming directly after slot creation”-functionality and perhaps that could work with either patch.

//Jesper


Sent from my iPhone

> On 16 Nov 2026, at 13:09, Andrey Borodin  wrote:
> Hi Magnus, all,
> 
> Nick reported the same startpoint/slot race in August [0], tracked in
> CF 7233 [1]. The client patch there opens the streaming connection and
> creates its slot before BASE_BACKUP, but starts streaming only after
> receiving the backup startpoint. Your proposal goes further than that.
> 
> We also discussed cleanup of a newly created permanent slot when
> BASE_BACKUP refuses to start. Zsolt's reports of slot invalidation and
> idle timeout are relevant to that approach too.
> 
> Could we combine the work and keep both threads linked from one CF
> entry? This thread is currently tracked separately as CF 7613 [2].
> 
> Thank you!
> 
> Best regards, Andrey Borodin.
> 
> [0] https://www.postgresql.org/message-id/CALP_NYTTvKzJ+chEUXLsyGaM+HYcP1WwGwXkZEnDfv-7FbPowQ@mail.gmail.com
> [1] https://commitfest.postgresql.org/patch/7233/
> [2] https://commitfest.postgresql.org/patch/7613/


FAQ