Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

Поиск
Список
Период
Сортировка
Искать

Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
"Alex Speller" <alex@alexspeller.com>
Дата:
Вложения:
Hi,

Since bc971f4025c (16), a physical walsender can sleep with flushed WAL it hasn't sent. With a single synchronous standby, the committing backend then waits in SyncRep until the standby's next status message: up to wal_receiver_status_interval (10 s by default).

WalSndLoop() resets the latch, and XLogSendPhysical() reads GetFlushRecPtr() and finds nothing to send. Only later does WalSndWait() join wal_flush_cv with ConditionVariablePrepareToSleep(), and it goes straight into WaitEventSetWait() without testing again. A backend whose XLogFlush() completes in between calls WalSndWakeup(), and ConditionVariableBroadcast() sets only the latches of processes already on the waitlist, so the wakeup is lost. ConditionVariablePrepareToSleep()'s header comment requires the exit condition to be tested after preparing. Before 16, WalSndWakeup() set every walsender's latch, so a wakeup after ResetLatch() couldn't be lost.

The backend is released while the walsender processes a standby reply. The walsender then goes on to read the flush pointer and head for WalSndWait(), so a client that commits again straight away often hits the window.

We hit this on 18.6 on Linux, with one synchronous standby in another availability zone and one client committing small transactions: commits stalled for up to 8 s. A probe sampling the primary every 50 ms saw five stalls of 0.2-6.2 s in about 400k commits. Throughout each one, the commit's WAL (184 or 336 bytes from pg_current_wal_insert_lsn() to sent_lsn) was unsent, the walsender was idle in WalSenderMain and the backend was waiting in SyncRep. The stalls ended on the standby's 10 s status-message cycle. With 'ANY 1 (a, b)' and two standbys we saw none in about 2M commits.

Reproduction (attached repro.sh): a primary and one streaming standby on one host over unix sockets, synchronous_standby_names naming the standby, and one pgbench client running a one-row insert for 120 s. macOS/arm64 builds from source, plus the Debian package in a Linux container on the same host:

    build                              synchronous_commit  commits  stalls > 1 s (with the signature)
    15.14                              remote_write        186,164  0, max 124 ms
    16.15                              remote_write         89,909  8 (8), max 10.0 s
    REL_18_STABLE 19a2002b5db          remote_write        158,943  11 (10), max 10.0 s
    REL_18_STABLE 19a2002b5db          on                  104,759  14 (12), max 10.0 s
    REL_18_STABLE + patch              on                  434,509  4 (0)
    master 57926d757ab + patch, 300 s  remote_write        388,146  2 (0)
    18.6-1.pgdg13+2, Linux             remote_write        101,218  4 (3), max 10.0 s

The signature, sampled every 50 ms for the whole stall: 176-432 bytes flushed but unsent, nothing unwritten or unflushed on the standby, the walsender in WalSenderMain and the client in IPC:SyncRep. The other stalls were host pauses, in which neither server's sampler produced any rows.

To check the mechanism directly, a test-only pg_usleep(20 ms) after send_data() in WalSndLoop() (physical, caught up; attached test-race-delay.patch) widens the window. At synchronous_commit = on in 60 s, unpatched completes 14 commits, 9 of them stalling for 4.5-10.0 s. With the fix it completes 1,194, with a maximum of 77 ms.

The attached patch rechecks after joining the waitlist. In WalSndWait(), a physical walsender in its main loop that is caught up with nothing pending compares GetFlushRecPtr() (GetStandbyFlushRecPtr() when cascading) with sentPtr after ConditionVariablePrepareToSleep(), and returns without sleeping if there is WAL to send. An alternative would be to join the CV before send_data() reads the flush pointer; I haven't tried that.

Attached: 0001-Recheck-for-WAL-after-a-physical-walsender-joins-wal.patch (applies to master 6cda9dfc4f9 and REL_18_STABLE), test-race-delay.patch (test only, not for commit), repro.sh and analyse.py.

WalSndWaitForWal() (logical) reads the flush pointer before WalSndWait() in the same way. I couldn't reproduce a stall with a synchronous subscription and haven't looked further.

Setting wal_receiver_status_interval = 1 on the standby caps the stall at about 1 s.

Thanks,
Alex

Re: Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
Вложения:
On Mon, Nov 2, 2026 at 7:44 PM Alex Speller  wrote:
> The attached patch rechecks after joining the waitlist.

Thanks for the report and patch!
The patch looks good to me, except for one comment.

Cascading walsenders also seem to miss wakeups for promotion or
a timeline switch, even when the flush position has not advanced.
Is that right? If so, I think we should recheck those conditions after
joining the waitlist as well.

I've attached an updated patch with these checks and a few
cosmetic changes. Thought?

> WalSndWaitForWal() (logical) reads the flush pointer before WalSndWait() in the same way.

Yes, so logical replication seems to have a similar issue.
I'll investigate that separately.

Regards,

-- 
Fujii Masao

Re: Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
alex@alexspeller.com
Дата:
On Mon, Nov 2, 2026 at 4:05 PM UTC, Fujii Masao <masao.fujii@gmail.com> wrote:
> Cascading walsenders also seem to miss wakeups for promotion or
> a timeline switch, even when the flush position has not advanced.
> Is that right? If so, I think we should recheck those conditions after
> joining the waitlist as well.

Yes. At promotion GetStandbyFlushRecPtr() doesn't move, because the end-of-recovery record is inserted, not replayed. That record's XLogFlush() also wakes the walsenders while RecoveryInProgress() is still true, so a cascading walsender is often between XLogSendPhysical() and WalSndWait() when StartupXLOG() calls WalSndWakeup(). The timeline-switch wakeup in ApplyWalRecord() has the same shape.

To check, I promoted a cascading standby (primary -> s1 -> s2, promoting s1) with test-race-delay.patch (200 ms) on s1. On REL_18_STABLE with v1, the walsender slept through the promotion in 39 of 40 rounds and only noticed when something else woke it; s2 was over a second late in 4. With v2 none of 70 rounds were late, and in the 30 I logged the new check caught the promotion every time. On master v1 and v2 looked the same at wal_level = replica, because the ProcSignalBarrier in UpdateLogicalDecodingStatusEndOfRecovery() sets the walsender's latch. At wal_level = logical, v1 was over a second late in 13 of 20 rounds and v2 in none.

> I've attached an updated patch with these checks and a few
> cosmetic changes. Thought?

v2 looks good to me. With assertions on master and REL_16/17/18_STABLE (the PG16 version on 16), the recovery TAP tests pass and the repro has no commit over 100 ms. With test-race-delay.patch (20 ms) on the primary at synchronous_commit = on, master, 16 and 18 each complete about 1,200 commits in 60 s, none over 91 ms.

Thanks,

Alex




Re: Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Tue, Nov 3, 2026 at 6:23 AM  wrote:
> v2 looks good to me.

Thanks for the review and testing!
I've pushed the patch.

Regards,

-- 
Fujii Masao


Re: Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
Вложения:
On Tue, Nov 3, 2026 at 1:05 AM Fujii Masao  wrote:
> > WalSndWaitForWal() (logical) reads the flush pointer before WalSndWait() in the same way.
>
> Yes, so logical replication seems to have a similar issue.
> I'll investigate that separately.

Probably I was able to reproduce the same issue in logical replication
by adding a sleep just before WalSndWaitForWal() calls WalSndWait().

WalSndWaitForWal() checks the flush or replay position before
WalSndWait() joins the waitlist. If the required WAL becomes available
between these steps, the notification can be lost, leaving the sender
asleep even though enough WAL is available to continue decoding.

I've attached a patch that rechecks the position after joining the waitlist
and skips the wait if enough WAL is available.

Regards,

-- 
Fujii Masao

Re: Lost walsender wakeup stalls synchronous commits until the standby's next reply (16+)

От:
Xuneng Zhou <xunengzhou@gmail.com>
Дата:
Вложения:
On Thu, Nov 12, 2026 at 9:42 AM Fujii Masao  wrote:
>
> On Tue, Nov 3, 2026 at 1:05 AM Fujii Masao  wrote:
> > > WalSndWaitForWal() (logical) reads the flush pointer before WalSndWait() in the same way.
> >
> > Yes, so logical replication seems to have a similar issue.
> > I'll investigate that separately.
>
> Probably I was able to reproduce the same issue in logical replication
> by adding a sleep just before WalSndWaitForWal() calls WalSndWait().
>
> WalSndWaitForWal() checks the flush or replay position before
> WalSndWait() joins the waitlist. If the required WAL becomes available
> between these steps, the notification can be lost, leaving the sender
> asleep even though enough WAL is available to continue decoding.
>
> I've attached a patch that rechecks the position after joining the waitlist
> and skips the wait if enough WAL is available.

Thanks for the patch! v1 looks correct for WalSenderWaitForWal.

It seems that WaitForStandbyConfirmation (17+) has the same race: a
standby can confirm between NeedToWaitForStandbys() and joining
wal_confirm_rcv_cv. Later replies would not wake it until restart_lsn
advances, which could stall synchronous commits too.

The attached reproducer adds a 3s delay before joining the waitlist,
and resumes the standby during that delay. With wal_sender_timeout = 0
and pg_recvlogical -s 0, the sender stalls for about 10s, its idle
timeout, even if the standby has caught up.

If this is an issue, do we need to apply the recheck like
StandbySlotsHaveCaughtup() before registration as well?

--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
FAQ