Обсуждение: pg_stat_replication.*_lag sometimes shows NULL during active replication

Hi hackers,

I have noticed that pg_stat_replication.*_lag sometimes shows NULL
when inserting a record per second for health checking. This happens
when the startup process replays WAL fast enough before the
walreceiver sends its flush notification to the walsender.

Here is the sequence that triggers the issue: (See normal.svg and
error.svg for diagrams of the normal and problematic cases.)

1. The walreceiver receives, writes, and flushes WAL, then wakes the
startup process via WakeupRecovery().

2. The startup process replays all available WAL quickly, then calls
WalRcvForceReply() to set force_reply = true and wakes the
walreceiver.

3. The walreceiver sends a flush notification to the walsender
(XLogWalRcvSendReply() in XLogWalRcvFlush()). Since the startup has
already replayed the WAL by this point, this message reports the
incremented applyPtr, which equals sentPtr. The walsender processes
this message, consuming the LagTracker samples and setting
fullyAppliedLastTime = true.

4. In the next loop iteration, the walreceiver sees force_reply = true
and sends another reply with the same positions. The walsender sees
applyPtr == sentPtr for the second consecutive time and sets
clearLagTimes = true. Since the LagTracker samples were already
consumed by step 3, all lag values are -1. With clearLagTimes = true,
these -1 values are written to walsnd->*Lag, causing
pg_stat_replication to show NULL.

The comment in ProcessStandbyReplyMessage() says:

     * If the standby reports that it has fully replayed the WAL in two
     * consecutive reply messages, then the second such message must result
     * from wal_receiver_status_interval expiring on the standby.

But as shown above, the second message can also come from
WalRcvForceReply(), violating this assumption.

The attached patch fixes this by adding a check that all lag values
are -1 to the clearLagTimes condition. This ensures that clearLagTimes
only triggers when there are truly no new lag samples in two
consecutive messages (i.e., the system is genuinely idle), and not
when the samples were simply consumed by a preceding message in a
burst of replies.

Regards,


-- 
Best regards,
Shinya Kato
NTT OSS Center

On Tue, Feb 24, 2026 at 3:54 PM Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> Hi hackers,
>
> I have noticed that pg_stat_replication.*_lag sometimes shows NULL
> when inserting a record per second for health checking. This happens
> when the startup process replays WAL fast enough before the
> walreceiver sends its flush notification to the walsender.
>
> Here is the sequence that triggers the issue: (See normal.svg and
> error.svg for diagrams of the normal and problematic cases.)
>
> 1. The walreceiver receives, writes, and flushes WAL, then wakes the
> startup process via WakeupRecovery().
>
> 2. The startup process replays all available WAL quickly, then calls
> WalRcvForceReply() to set force_reply = true and wakes the
> walreceiver.
>
> 3. The walreceiver sends a flush notification to the walsender
> (XLogWalRcvSendReply() in XLogWalRcvFlush()). Since the startup has
> already replayed the WAL by this point, this message reports the
> incremented applyPtr, which equals sentPtr. The walsender processes
> this message, consuming the LagTracker samples and setting
> fullyAppliedLastTime = true.
>
> 4. In the next loop iteration, the walreceiver sees force_reply = true
> and sends another reply with the same positions. The walsender sees
> applyPtr == sentPtr for the second consecutive time and sets
> clearLagTimes = true. Since the LagTracker samples were already
> consumed by step 3, all lag values are -1. With clearLagTimes = true,
> these -1 values are written to walsnd->*Lag, causing
> pg_stat_replication to show NULL.
>
> The comment in ProcessStandbyReplyMessage() says:
>
>      * If the standby reports that it has fully replayed the WAL in two
>      * consecutive reply messages, then the second such message must result
>      * from wal_receiver_status_interval expiring on the standby.
>
> But as shown above, the second message can also come from
> WalRcvForceReply(), violating this assumption.
>
> The attached patch fixes this by adding a check that all lag values
> are -1 to the clearLagTimes condition. This ensures that clearLagTimes
> only triggers when there are truly no new lag samples in two
> consecutive messages (i.e., the system is genuinely idle), and not
> when the samples were simply consumed by a preceding message in a
> burst of replies.

Thanks for the patch!

With the patch applied, I set up a logical replication and inserted a row every
second. Even with continuous inserts, NULL was shown in the lag columns of
pg_stat_replication. That makes me wonder whether the patch's approach is
sufficient to address the issue.

Relying solely on replies from the standby or subscriber seems a bit fragile to
me. If the goal is to keep showing the last measured lag for some time,
perhaps we should introduce a rate limit on when NULL is displayed in the lag
columns?

For example, if there has been no activity (i.e., sentPtr == applyPtr and
applyPtr has not changed since the previous cycle) for, say, 10 seconds,
then we could allow NULL to be shown. Thought?

Regards,

--
Fujii Masao

On Mon, Mar 2, 2026 at 11:44 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> With the patch applied, I set up a logical replication and inserted a row every
> second. Even with continuous inserts, NULL was shown in the lag columns of
> pg_stat_replication. That makes me wonder whether the patch's approach is
> sufficient to address the issue.

Thank you for the review and testing! I had only considered the issue
in the context of physical replication, but as you pointed out, my
approach is insufficient for logical replication.

> Relying solely on replies from the standby or subscriber seems a bit fragile to
> me. If the goal is to keep showing the last measured lag for some time,
> perhaps we should introduce a rate limit on when NULL is displayed in the lag
> columns?

My primary goal was to ensure that the source code comments match the
actual behavior, as the comment stating "the second such message must
result from wal_receiver_status_interval expiring on the standby" is
inaccurate. However, as you noted, the patch alone is not sufficient
to fully address the issue.

> For example, if there has been no activity (i.e., sentPtr == applyPtr and
> applyPtr has not changed since the previous cycle) for, say, 10 seconds,
> then we could allow NULL to be shown. Thought?

I considered a time-based rate limit, but it is difficult to choose an
appropriate threshold. Furthermore, the walsender has no way of
knowing the standby's or subscriber's wal_receiver_status_interval
setting.

The attached v2 patch takes a different approach: it additionally
requires that all reported positions (write/flush/apply) remain
unchanged from the previous reply. This directly detects a truly idle
system without relying on timeouts—if any position has advanced, new
WAL activity must have occurred, so we should not clear the lag values
even if the lag tracker is empty.
--
Best regards,
Shinya Kato
NTT OSS Center

On Fri, Mar 6, 2026 at 4:13 PM Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> On Mon, Mar 2, 2026 at 11:44 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> > With the patch applied, I set up a logical replication and inserted a row every
> > second. Even with continuous inserts, NULL was shown in the lag columns of
> > pg_stat_replication. That makes me wonder whether the patch's approach is
> > sufficient to address the issue.
>
> Thank you for the review and testing! I had only considered the issue
> in the context of physical replication, but as you pointed out, my
> approach is insufficient for logical replication.
>
> > Relying solely on replies from the standby or subscriber seems a bit fragile to
> > me. If the goal is to keep showing the last measured lag for some time,
> > perhaps we should introduce a rate limit on when NULL is displayed in the lag
> > columns?
>
> My primary goal was to ensure that the source code comments match the
> actual behavior, as the comment stating "the second such message must
> result from wal_receiver_status_interval expiring on the standby" is
> inaccurate. However, as you noted, the patch alone is not sufficient
> to fully address the issue.
>
> > For example, if there has been no activity (i.e., sentPtr == applyPtr and
> > applyPtr has not changed since the previous cycle) for, say, 10 seconds,
> > then we could allow NULL to be shown. Thought?
>
> I considered a time-based rate limit, but it is difficult to choose an
> appropriate threshold. Furthermore, the walsender has no way of
> knowing the standby's or subscriber's wal_receiver_status_interval
> setting.
>
> The attached v2 patch takes a different approach: it additionally
> requires that all reported positions (write/flush/apply) remain
> unchanged from the previous reply. This directly detects a truly idle
> system without relying on timeouts—if any position has advanced, new
> WAL activity must have occurred, so we should not clear the lag values
> even if the lag tracker is empty.

This approach looks good to me.

One comment: currently, the lag becomes NULL basically after about one
wal_receiver_status_interval during periods of no activity. OTOH, with this
approach, it seems it would take about twice wal_receiver_status_interval.
Is this understanding correct?

Regards,

--
Fujii Masao

On Mon, Mar 9, 2026 at 8:21 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> > The attached v2 patch takes a different approach: it additionally
> > requires that all reported positions (write/flush/apply) remain
> > unchanged from the previous reply. This directly detects a truly idle
> > system without relying on timeouts—if any position has advanced, new
> > WAL activity must have occurred, so we should not clear the lag values
> > even if the lag tracker is empty.
>
> This approach looks good to me.

Thank you for looking into this.

> One comment: currently, the lag becomes NULL basically after about one
> wal_receiver_status_interval during periods of no activity. OTOH, with this
> approach, it seems it would take about twice wal_receiver_status_interval.
> Is this understanding correct?

Exactly. With this patch, it takes about two
wal_receiver_status_interval cycles to show NULL instead of one. I
think this is an acceptable trade-off because it is better to take a
bit longer to detect inactivity than to incorrectly show NULL during
active replication.

--
Best regards,
Shinya Kato
NTT OSS Center

On Tue, Mar 10, 2026 at 10:02 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> On Mon, Mar 9, 2026 at 8:21 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> > > The attached v2 patch takes a different approach: it additionally
> > > requires that all reported positions (write/flush/apply) remain
> > > unchanged from the previous reply. This directly detects a truly idle
> > > system without relying on timeouts—if any position has advanced, new
> > > WAL activity must have occurred, so we should not clear the lag values
> > > even if the lag tracker is empty.
> >
> > This approach looks good to me.
>
> Thank you for looking into this.
>
> > One comment: currently, the lag becomes NULL basically after about one
> > wal_receiver_status_interval during periods of no activity. OTOH, with this
> > approach, it seems it would take about twice wal_receiver_status_interval.
> > Is this understanding correct?
>
> Exactly. With this patch, it takes about two
> wal_receiver_status_interval cycles to show NULL instead of one. I
> think this is an acceptable trade-off because it is better to take a
> bit longer to detect inactivity than to incorrectly show NULL during
> active replication.

Even with your latest patch, if we remove fullyAppliedLastTime, and set
clearLagTimes to true when applyPtr == sentPtr && noLagSamples &&
positionsUnchanged,
wouldn't the time for the lag to become NULL be almost the same as
wal_receiver_status_interval?

The documentation doesn't clearly specify how long it should take for
the lag to become NULL, so doubling that time might be acceptable.
However, if we can keep it roughly the same without much complexity,
I think that would be preferable.

Thought?

--
Fujii Masao

On Tue, Mar 10, 2026 at 10:54 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> Even with your latest patch, if we remove fullyAppliedLastTime, and set
> clearLagTimes to true when applyPtr == sentPtr && noLagSamples &&
> positionsUnchanged,
> wouldn't the time for the lag to become NULL be almost the same as
> wal_receiver_status_interval?
>
> The documentation doesn't clearly specify how long it should take for
> the lag to become NULL, so doubling that time might be acceptable.
> However, if we can keep it roughly the same without much complexity,
> I think that would be preferable.
>
> Thought?

Thank you for the suggestion. I tested this by removing
fullyAppliedLastTime, but even with synchronous replication, NULL
still appears. Here is why:

- Reply 1 (flush notification): positions = X. Lag samples are
consumed with real values, so noLagSamples = false. clearLagTimes is
not set, and prevPtrs = X is saved.

- Reply 2 (force_reply): positions = X again. Here, noLagSamples =
true and positionsUnchanged = true. Since applyPtr == sentPtr,
clearLagTimes is set to true, resulting in a NULL value.

Therefore, I believe fullyAppliedLastTime is still necessary to ensure
that the previous reply also contained no lag samples.

BTW I noticed an incorrect comment in walreceiver.c and have included
a fix for it. Patch 0001 remains unchanged.


--
Best regards,
Shinya Kato
NTT OSS Center

On Wed, Mar 11, 2026 at 11:39 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> On Tue, Mar 10, 2026 at 10:54 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> > Even with your latest patch, if we remove fullyAppliedLastTime, and set
> > clearLagTimes to true when applyPtr == sentPtr && noLagSamples &&
> > positionsUnchanged,
> > wouldn't the time for the lag to become NULL be almost the same as
> > wal_receiver_status_interval?
> >
> > The documentation doesn't clearly specify how long it should take for
> > the lag to become NULL, so doubling that time might be acceptable.
> > However, if we can keep it roughly the same without much complexity,
> > I think that would be preferable.
> >
> > Thought?
>
> Thank you for the suggestion. I tested this by removing
> fullyAppliedLastTime, but even with synchronous replication, NULL
> still appears. Here is why:
>
> - Reply 1 (flush notification): positions = X. Lag samples are
> consumed with real values, so noLagSamples = false. clearLagTimes is
> not set, and prevPtrs = X is saved.
>
> - Reply 2 (force_reply): positions = X again. Here, noLagSamples =
> true and positionsUnchanged = true. Since applyPtr == sentPtr,
> clearLagTimes is set to true, resulting in a NULL value.
>
> Therefore, I believe fullyAppliedLastTime is still necessary to ensure
> that the previous reply also contained no lag samples.

Thanks for testing and for the clarification! You're right.

However, if we apply this change, the time required for the lag information to
be reset would effectively double. I start wondering if that's really
acceptable, especially for back branches. Although the docs doesn't clearly
specify this timing, doubling it could affect systems that monitor
replication lag, for example. It might still be reasonable to apply
such a change in master, though.

On further thought, the root cause seems to be that walreceiver can send
two consecutive status reply messages with identical WAL locations even
when wal_receiver_status_interval has not yet elapsed. Addressing that
behavior directly might resolve the issue you reported. I've attached a PoC
patch that does this. Thought?

Regards,

--
Fujii Masao

> On Mar 12, 2026, at 23:27, Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Wed, Mar 11, 2026 at 11:39 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
>>
>> On Tue, Mar 10, 2026 at 10:54 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>>> Even with your latest patch, if we remove fullyAppliedLastTime, and set
>>> clearLagTimes to true when applyPtr == sentPtr && noLagSamples &&
>>> positionsUnchanged,
>>> wouldn't the time for the lag to become NULL be almost the same as
>>> wal_receiver_status_interval?
>>>
>>> The documentation doesn't clearly specify how long it should take for
>>> the lag to become NULL, so doubling that time might be acceptable.
>>> However, if we can keep it roughly the same without much complexity,
>>> I think that would be preferable.
>>>
>>> Thought?
>>
>> Thank you for the suggestion. I tested this by removing
>> fullyAppliedLastTime, but even with synchronous replication, NULL
>> still appears. Here is why:
>>
>> - Reply 1 (flush notification): positions = X. Lag samples are
>> consumed with real values, so noLagSamples = false. clearLagTimes is
>> not set, and prevPtrs = X is saved.
>>
>> - Reply 2 (force_reply): positions = X again. Here, noLagSamples =
>> true and positionsUnchanged = true. Since applyPtr == sentPtr,
>> clearLagTimes is set to true, resulting in a NULL value.
>>
>> Therefore, I believe fullyAppliedLastTime is still necessary to ensure
>> that the previous reply also contained no lag samples.
>
> Thanks for testing and for the clarification! You're right.
>
> However, if we apply this change, the time required for the lag information to
> be reset would effectively double. I start wondering if that's really
> acceptable, especially for back branches. Although the docs doesn't clearly
> specify this timing, doubling it could affect systems that monitor
> replication lag, for example. It might still be reasonable to apply
> such a change in master, though.
>
> On further thought, the root cause seems to be that walreceiver can send
> two consecutive status reply messages with identical WAL locations even
> when wal_receiver_status_interval has not yet elapsed. Addressing that
> behavior directly might resolve the issue you reported. I've attached a PoC
> patch that does this. Thought?
>
> Regards,
>
> --
> Fujii Masao
> <v4-0001-Avoid-sending-duplicate-WAL-locations-in-standby-.patch>

I just read v4. The approach looks good to me overall. I have a few comments about the naming.

This patch changes the old force reply logic to an applied-location-driven reply. Now a reply is sent only if the
appliedlocation has advanced. However, this applied-location-driven reply is still triggered from WalRcvForceReply(),
sothe function has effectively lost its original “force” semantics. Because of that, it might be better to rename
WalRcvForceReply()to something like WalRcvRequestApplyStatusUpdate(). 

Then,
```
static void
XLogWalRcvSendReply(bool force, bool requestReply, bool replyApply)
```

replyApply reads like “send an apply reply”, but in reality it indicates that the applied location should be checked to
decidewhether to send the reply. So it might be clearer to rename it to something like checkApplyStatus. 

Lastly,
```
    sig_atomic_t reply_apply; /* used as a bool */
```

reply_apply sounds like an action of “reply with apply”, but what it actually represents is that the startup process
requestedan applied-location-driven reply. If applied location is not advanced, the reply won’t be sent. So a name like
apply_update_requestedmight better reflect the meaning. 

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

On Fri, Mar 13, 2026 at 12:27 AM Fujii Masao <masao.fujii@gmail.com> wrote:

> Thanks for testing and for the clarification! You're right.
>
> However, if we apply this change, the time required for the lag information to
> be reset would effectively double. I start wondering if that's really
> acceptable, especially for back branches. Although the docs doesn't clearly
> specify this timing, doubling it could affect systems that monitor
> replication lag, for example. It might still be reasonable to apply
> such a change in master, though.

Yes, I agree. Doubling the lag reset time should be avoided in back
branches if possible.

> On further thought, the root cause seems to be that walreceiver can send
> two consecutive status reply messages with identical WAL locations even
> when wal_receiver_status_interval has not yet elapsed. Addressing that
> behavior directly might resolve the issue you reported. I've attached a PoC
> patch that does this. Thought?

Thank you for the v4 patch. I think this approach is better than mine.
I tested the patch and confirmed that the issue no longer reproduces
with physical replication. However, with logical replication, the lag
columns in pg_stat_replication still show NULL periodically at
wal_receiver_status_interval, since send_feedback() in worker.c can
still send duplicate positions.

+ * previsou update, i.e., when 'replyApply' is true.

One minor thing: there is a typo "previsou". It should be "previous".


--
Best regards,
Shinya Kato
NTT OSS Center

On Mon, Mar 16, 2026 at 9:26 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
> Thank you for the v4 patch. I think this approach is better than mine.
> I tested the patch and confirmed that the issue no longer reproduces
> with physical replication. However, with logical replication, the lag
> columns in pg_stat_replication still show NULL periodically at
> wal_receiver_status_interval, since send_feedback() in worker.c can
> still send duplicate positions.

I was thinking that if a feedback message triggered by
wal_receiver_status_interval has the same LSNs as the previous message,
it's expected for the lag columns to become NULL. But you see it differently,
don't you? Sorry, I failed to understand your point...

Regards,

--
Fujii Masao

On Tue, Mar 17, 2026 at 11:00 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Mon, Mar 16, 2026 at 9:26 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
> > Thank you for the v4 patch. I think this approach is better than mine.
> > I tested the patch and confirmed that the issue no longer reproduces
> > with physical replication. However, with logical replication, the lag
> > columns in pg_stat_replication still show NULL periodically at
> > wal_receiver_status_interval, since send_feedback() in worker.c can
> > still send duplicate positions.
>
> I was thinking that if a feedback message triggered by
> wal_receiver_status_interval has the same LSNs as the previous message,
> it's expected for the lag columns to become NULL. But you see it differently,
> don't you? Sorry, I failed to understand your point...

Sorry for the confusion. I ran a script inserting one row every 0.5
seconds under logical replication and confirmed that NULL still
appears in the lag columns even while replication is actively running.
I was initially mistaken that this was tied to
wal_receiver_status_interval timing — that turned out to be unrelated.

I haven't had time to investigate further, but my current impression
is that the existing approach may not be sufficient for logical
replication.


--
Best regards,
Shinya Kato
NTT OSS Center

On Thu, Mar 19, 2026 at 10:58 PM Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> On Tue, Mar 17, 2026 at 11:00 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> >
> > On Mon, Mar 16, 2026 at 9:26 AM Shinya Kato <shinya11.kato@gmail.com> wrote:
> > > Thank you for the v4 patch. I think this approach is better than mine.
> > > I tested the patch and confirmed that the issue no longer reproduces
> > > with physical replication. However, with logical replication, the lag
> > > columns in pg_stat_replication still show NULL periodically at
> > > wal_receiver_status_interval, since send_feedback() in worker.c can
> > > still send duplicate positions.
> >
> > I was thinking that if a feedback message triggered by
> > wal_receiver_status_interval has the same LSNs as the previous message,
> > it's expected for the lag columns to become NULL. But you see it differently,
> > don't you? Sorry, I failed to understand your point...
>
> Sorry for the confusion. I ran a script inserting one row every 0.5
> seconds under logical replication and confirmed that NULL still
> appears in the lag columns even while replication is actively running.
> I was initially mistaken that this was tied to
> wal_receiver_status_interval timing — that turned out to be unrelated.
>
> I haven't had time to investigate further, but my current impression
> is that the existing approach may not be sufficient for logical
> replication.

Thanks for the clarification! I understand your point now.

I think the issue occurs when the positions in the first message point to
the same LSN (e.g., 0/030D5230), and the second message reports the same but
larger LSN (e.g., 0/030D52E0).

I've updated the patch to address this. It removes fullyAppliedLastTime,
tracks the positions from the previous reply, and clears the lag values only
when the positions remain unchanged across two consecutive messages.

Patch attached. Could you test and review this updated patch?

Regards,

--
Fujii Masao

On Fri, Mar 20, 2026 at 2:13 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> I think the issue occurs when the positions in the first message point to
> the same LSN (e.g., 0/030D5230), and the second message reports the same but
> larger LSN (e.g., 0/030D52E0).

Thanks for the explanation!

> I've updated the patch to address this. It removes fullyAppliedLastTime,
> tracks the positions from the previous reply, and clears the lag values only
> when the positions remain unchanged across two consecutive messages.
>
> Patch attached. Could you test and review this updated patch?

The patch works properly. I think it looks nice to me, except for the
typo I sent in the previous message.


--
Best regards,
Shinya Kato
NTT OSS Center