Re: walsummarizer can get stuck when switching timelines

Re: walsummarizer can get stuck when switching timelines

> On 14 Jul 2026, at 01:39, Robert Haas  wrote:
> 
> Here is a patch, with a test case.


Hi Robert!

I read the patch and I think it's correct. Two small things on the new
fallback. First, the test covers a single switch; the tricky case is several
switches within one segment, where the fallback loop and the switch_lsn bound
interact per timeline. It seems correct to me: we stop at switch_lsn, so we
only read the pre-switch bytes both timelines share. But it might be worth a
test. Second, in the fallback path WALRead reopens the segment on every page,
because the requested tli never matches seg.ws_tli once
summarizer_wal_segment_open() has moved to a descendant. These open/close
might be an exatra overhead?

I think, there's a symmetric bug on the recovery side with the same root cause.
Your patch makes the summarizer fall forward to a descendant timeline, which
looks safe. XLogFileReadAnyTLI() does the opposite: when the switch-point segment
is missing on the target timeline, it falls back to an ancestor and reads the
same segno. But that segment holds divergent data continuing old timeline!
Recovery applies it silently and then can't reach the intended timeline.

The comment you added to t/003_tli_switch.pl states the assumption that breaks
here:

  # ... recovery will read them from there and work just fine.

That holds only when the new timeline's segment is already there. Concretely,
with just two timelines: TL2 forks from TL1 in segment 67, which is archived
for TL1 but not yet for TL2. Recovery targets TL2 and asks for segment 67;
TL2's copy isn't in the archive, so XLogFileReadAnyTLI() falls back to TL1 and
hands back TL1's segment 67. Past the switch point that is the old primary's
divergent WAL, recovery applies it, and now it can never get onto TL2. The
beginseg check doesn't help here - it only skips a timeline for segments older
than that timeline's start, so nothing stops the fallback to the ancestor TL1
for the very segment where the two diverge. I reported it with a fix [0].


Regards, Andrey Borodin.

[0] https://postgr.es/m/85386EF6-16B7-4D62-86BE-526A10F93825%40yandex-team.ru (CF #6525)

Re: walsummarizer can get stuck when switching timelines

> On 14 Jul 2026, at 17:49, Amit Kapila  wrote:
> 
> I tried the patch on Windows and got following failure:

Hi Amit,

Thanks for testing! That output is exactly what an unpatched summarizer produces
on my machine. I suspect the fix didn't make it into your build.
Also the test passes for me on Windows CI with the patch applied [0].

Could you double-check that you observe the failure?


Best regards, Andrey Borodin.

[0] https://github.com/x4m/postgres_g/actions/runs/29330084058/job/87076116050

Re: walsummarizer can get stuck when switching timelines

> On 16 Jul 2026, at 17:19, Robert Haas  wrote:
> 
> In the scenario tested by this test case, recovery works fine. Without
> the patch, node3 follows the timeline switch from TLI 1 to TLI 2 and
> continues recovery thereafter without a problem. Only WAL
> summarization gets stuck. Maybe I need to go read your thread to
> figure out why you think these are the same problem, but based on what
> the test case does, I feel like they are different problems.

I toyed a bit with your test and it is safe as long as node1 has is own
archive. So yes, technically there's no problem.

When node3 and node1 has shared archive some write activity triggers
problematic behavior.

PFA demo test to illustrate this. First step is your v1 patch intact.
Step 2 is test change to trigger the problem.

Anyway, this happens only when nodes have common archive.


Best regards, Andrey Borodin.

Re: walsummarizer can get stuck when switching timelines

> On 16 Jul 2026, at 01:18, Robert Haas  wrote:
> 
> 
> understand Andrey's probable desire to have me review his patch on
> that thread, but I think whatever is going on there is only
> tangentially related to this problem.

Of course I'd like that fix committed too :) But that's not why I'm raising it here.

+# The reason for the problem is that when a new primary is promoted, the
+# partial file that ends the old timeline is renamed, giving it a ".partial"
+# suffix, and is not archived. That's not a problem for recovery, because
+# the bytes that appear at the start of that segment will be copied into the
+# first segment on the new timeline, and recovery will read them from there
+# and work just fine.

That only holds when the new timeline's segment is already available. If it isn't,
XLogFileReadAnyTLI() falls back to the ancestor's segment and, past the switch point,
hands recovery the old timeline's divergent WAL. It's the mirror, on the recovery
side, of what your patch fixes for the summarizer.

I'm not asking you to fix XLogFileReadAnyTLI() here. I just don't think the new
comment should claim recovery is unaffected.


Best regards, Andrey Borodin.

Re: walsummarizer can get stuck when switching timelines

On Wed, Jul 15, 2026 at 1:43 PM Srinath Reddy Sadipiralla
 wrote:
> a nitpick in the 003_tli_switch.pl file the comment says
> "....giving it a ".partial" suffix, and is not archived," but AFAIK,
> We do archive the .partial files right? CleanupAfterArchiveRecovery
> renames and notifies the archiver, or am I reading it wrong?

You're right. I'll reword the comment.

> +1

I think this is all a digression from the problem at hand. I
understand Andrey's probable desire to have me review his patch on
that thread, but I think whatever is going on there is only
tangentially related to this problem.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Thu, Jul 16, 2026 at 2:05 AM Andrey Borodin  wrote:
> Of course I'd like that fix committed too :) But that's not why I'm raising it here.
>
> +# The reason for the problem is that when a new primary is promoted, the
> +# partial file that ends the old timeline is renamed, giving it a ".partial"
> +# suffix, and is not archived. That's not a problem for recovery, because
> +# the bytes that appear at the start of that segment will be copied into the
> +# first segment on the new timeline, and recovery will read them from there
> +# and work just fine.
>
> That only holds when the new timeline's segment is already available. If it isn't,
> XLogFileReadAnyTLI() falls back to the ancestor's segment and, past the switch point,
> hands recovery the old timeline's divergent WAL. It's the mirror, on the recovery
> side, of what your patch fixes for the summarizer.
>
> I'm not asking you to fix XLogFileReadAnyTLI() here. I just don't think the new
> comment should claim recovery is unaffected.

In the scenario tested by this test case, recovery works fine. Without
the patch, node3 follows the timeline switch from TLI 1 to TLI 2 and
continues recovery thereafter without a problem. Only WAL
summarization gets stuck. Maybe I need to go read your thread to
figure out why you think these are the same problem, but based on what
the test case does, I feel like they are different problems.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Tue, Jul 14, 2026 at 8:41 AM Andrey Borodin  wrote:
> It seems correct to me: we stop at switch_lsn, so we
> only read the pre-switch bytes both timelines share. But it might be worth a
> test.

I don't think I'm sufficiently motivated to want to write a TAP test
for two timeline switches in the same segment. If you want to try it,
feel free. We can add that in a later commit; it doesn't need to be
part of this patch.

> Second, in the fallback path WALRead reopens the segment on every page,
> because the requested tli never matches seg.ws_tli once
> summarizer_wal_segment_open() has moved to a descendant. These open/close
> might be an exatra overhead?

I can believe this might be the case, and I'll look into fixing it. I
don't think it makes any practical difference because segments
containing timeline switches are rare and only contain so many blocks,
and opening and closing files is not that expensive. But it makes
sense to get this right.

> I think, there's a symmetric bug on the recovery side with the same root cause.
> Your patch makes the summarizer fall forward to a descendant timeline, which
> looks safe. XLogFileReadAnyTLI() does the opposite: when the switch-point segment
> is missing on the target timeline, it falls back to an ancestor and reads the
> same segno. But that segment holds divergent data continuing old timeline!
> Recovery applies it silently and then can't reach the intended timeline.

I think this is only a problem if somebody reads past the switchpoint.
In any case, this thread is not about fixing bugs in
XLogFileReadAnyTLI() or its callers.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Thu, Jul 16, 2026 at 7:46 AM Thom Brown  wrote:
> +# Cause the partial segment to get archived on the *new* timeline.
> +#
> +# In more detail: the WAL segment that contains the current insert
> LSN exists on
> +# timeline 1, but since all we did is CREATE TABLE dummy (), it
> wasn't full. We're
> +# now running on timeline 2, and pg_switch_wal() fills up the rest of
> the segment.
> +# So the full segment should get archived on timeline 2, but not on timeline 1.
> +# We do a CHECKPOINT here to make sure that the summarizer tries to progress.
> +$node2->safe_psql('postgres', < +SELECT pg_switch_wal();
> +CHECKPOINT;
>
> Wouldn't we want the checkpoint's redo record to make it into the
> archived segment *before* switching wal?

I don't think so, but maybe you want to explain why you think so.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin  wrote:
> I can reproduce it locally by adding an extra WAL switch on node1
> before promoting node2. It can probably be made deterministic either by
> polling pg_stat_archiver before relying on the segment (as
> 025_stuck_on_old_timeline.pl does) or by setting archive_mode=always on
> node2 (I would expect it to be a bit faster).

archive_mode=always seems like the correct fix. Here's v2.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Fri, Jul 17, 2026 at 12:22 PM Thom Brown  wrote:
> It's erroring out, but shouldn't it check for 000000020000000000000003
> before doing so? It wasn't removed, just renamed.

Actually, neither is the case. 000000010000000000000003 was not
archived. 000000020000000000000003 is a different file with partially
identical contents. But the fact that the existing error message here
is not quite right is not a problem this patch should be trying to
solve.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Thu, Jul 16, 2026 at 8:59 AM Thom Brown  wrote:
> Well if we run pg_switch_wal(), we're effectively closing the current
> segment. CHECKPOINT then emits its WAL after the switch. If the
> summarizer needs a CHECKPOINT_REDO record in the segment being
> archived, it would already be too late.

OK, I think this is where we're seeing things differently. The
summarizer doesn't need a CHECKPOINT_REDO record in the segment being
archived. Rather, we need a CHECKPOINT_REDO record somewhere after the
WAL that we want summarized. In this case, the WAL we want summarized
is generated by the pg_switch_wal() call.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: walsummarizer can get stuck when switching timelines

On Thu, 16 Jul 2026 at 13:39, Robert Haas  wrote:
>
> On Thu, Jul 16, 2026 at 7:46 AM Thom Brown  wrote:
> > +# Cause the partial segment to get archived on the *new* timeline.
> > +#
> > +# In more detail: the WAL segment that contains the current insert
> > LSN exists on
> > +# timeline 1, but since all we did is CREATE TABLE dummy (), it
> > wasn't full. We're
> > +# now running on timeline 2, and pg_switch_wal() fills up the rest of
> > the segment.
> > +# So the full segment should get archived on timeline 2, but not on timeline 1.
> > +# We do a CHECKPOINT here to make sure that the summarizer tries to progress.
> > +$node2->safe_psql('postgres', < > +SELECT pg_switch_wal();
> > +CHECKPOINT;
> >
> > Wouldn't we want the checkpoint's redo record to make it into the
> > archived segment *before* switching wal?
>
> I don't think so, but maybe you want to explain why you think so.

Well if we run pg_switch_wal(), we're effectively closing the current
segment. CHECKPOINT then emits its WAL after the switch. If the
summarizer needs a CHECKPOINT_REDO record in the segment being
archived, it would already be too late. I guess it works if the
checkpoint triggered by promotion always happens before
pg_switch_wal(), but it seems like it could just as well happen
afterwrds. If so, that would make the test depend on timing.

I could be misunderstanding the intended purpose here, though. It just
looked to me like CHECKPOINT; SELECT pg_switch_wal(); would be
deterministic, whereas the current ordering appears to rely on the
promotion checkpoint winning the race.

Or maybe I need to rethink after some caffeine.

Thom

Re: walsummarizer can get stuck when switching timelines

On Wed, 15 Jul 2026 at 21:19, Robert Haas  wrote:
>
> On Wed, Jul 15, 2026 at 1:43 PM Srinath Reddy Sadipiralla
>  wrote:
> > a nitpick in the 003_tli_switch.pl file the comment says
> > "....giving it a ".partial" suffix, and is not archived," but AFAIK,
> > We do archive the .partial files right? CleanupAfterArchiveRecovery
> > renames and notifies the archiver, or am I reading it wrong?
>
> You're right. I'll reword the comment.
>
> > +1
>
> I think this is all a digression from the problem at hand. I
> understand Andrey's probable desire to have me review his patch on
> that thread, but I think whatever is going on there is only
> tangentially related to this problem.

+# Cause the partial segment to get archived on the *new* timeline.
+#
+# In more detail: the WAL segment that contains the current insert
LSN exists on
+# timeline 1, but since all we did is CREATE TABLE dummy (), it
wasn't full. We're
+# now running on timeline 2, and pg_switch_wal() fills up the rest of
the segment.
+# So the full segment should get archived on timeline 2, but not on timeline 1.
+# We do a CHECKPOINT here to make sure that the summarizer tries to progress.
+$node2->safe_psql('postgres', <

Re: walsummarizer can get stuck when switching timelines

On Thu, 16 Jul 2026 at 14:45, Robert Haas  wrote:
>
> On Thu, Jul 16, 2026 at 8:59 AM Thom Brown  wrote:
> > Well if we run pg_switch_wal(), we're effectively closing the current
> > segment. CHECKPOINT then emits its WAL after the switch. If the
> > summarizer needs a CHECKPOINT_REDO record in the segment being
> > archived, it would already be too late.
>
> OK, I think this is where we're seeing things differently. The
> summarizer doesn't need a CHECKPOINT_REDO record in the segment being
> archived. Rather, we need a CHECKPOINT_REDO record somewhere after the
> WAL that we want summarized. In this case, the WAL we want summarized
> is generated by the pg_switch_wal() call.

Ah, gotcha. Sorry for the noise.

Thom

Re: walsummarizer can get stuck when switching timelines

On Thu, 16 Jul 2026 at 14:46, Thom Brown  wrote:
>
> On Thu, 16 Jul 2026 at 14:45, Robert Haas  wrote:
> >
> > On Thu, Jul 16, 2026 at 8:59 AM Thom Brown  wrote:
> > > Well if we run pg_switch_wal(), we're effectively closing the current
> > > segment. CHECKPOINT then emits its WAL after the switch. If the
> > > summarizer needs a CHECKPOINT_REDO record in the segment being
> > > archived, it would already be too late.
> >
> > OK, I think this is where we're seeing things differently. The
> > summarizer doesn't need a CHECKPOINT_REDO record in the segment being
> > archived. Rather, we need a CHECKPOINT_REDO record somewhere after the
> > WAL that we want summarized. In this case, the WAL we want summarized
> > is generated by the pg_switch_wal() call.
>
> Ah, gotcha. Sorry for the noise.

Another question; I'm seeing this in 003_tli_switch_node3.log:

2026-07-17 16:57:50.525 BST walsummarizer[714073] DEBUG:  timeline 1
became historic, can read up to 0/0301E118
2026-07-17 16:57:50.525 BST walsummarizer[714073] ERROR:  requested
WAL segment pg_wal/000000010000000000000003 has already been removed
2026-07-17 16:58:00.533 BST walsummarizer[714073] DEBUG:  switch point
from TLI 1 to TLI 2 is at 0/0301E118
2026-07-17 16:58:00.534 BST walsummarizer[714073] DEBUG:  summarized
WAL on TLI 1 from 0/02000028 to 0/0301E118
2026-07-17 16:58:00.534 BST walsummarizer[714073] DEBUG:  summarized
WAL on TLI 2 from 0/0301E118 to 0/0301E150

It's erroring out, but shouldn't it check for 000000020000000000000003
before doing so? It wasn't removed, just renamed.

Thom

Re: walsummarizer can get stuck when switching timelines

On Wed, Jul 15, 2026 at 7:46 PM Robert Haas  wrote:
>
> On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin  wrote:
> > I can reproduce it locally by adding an extra WAL switch on node1
> > before promoting node2. It can probably be made deterministic either by
> > polling pg_stat_archiver before relying on the segment (as
> > 025_stuck_on_old_timeline.pl does) or by setting archive_mode=always on
> > node2 (I would expect it to be a bit faster).
>
> archive_mode=always seems like the correct fix. Here's v2.
>

Thanks for the updated patch, it works for me.

-- 
With Regards,
Amit Kapila.

Re: walsummarizer can get stuck when switching timelines

On Tue, Jul 14, 2026 at 2:09 AM Robert Haas  wrote:
>
> Here is a patch, with a test case.
>

I tried the patch on Windows and got following failure:

# executing test in
C:\....\postgresql\build/testrun/pg_walsummary/003_tli_switch group
pg_walsummary test 003_tli_switch
# initializing database system by copying initdb template
not ok 1 - WAL summarization on node3 advanced past timeline switch
ok 2 - no summaries from before LSN 0/04016090
not ok 3 - at least one summary from LSN 0/04016090 or later
1..3
# test failed
----------------------------------- stderr -----------------------------------
# poll_query_until timed out executing this query:
# SELECT EXISTS (SELECT * FROM pg_available_wal_summaries() WHERE tli = 2)
#
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
#   Failed test 'WAL summarization on node3 advanced past timeline switch'
#   at C:/Workspace/code/postgresql/src/bin/pg_walsummary/t/003_tli_switch.pl
line 92.
#   Failed test 'at least one summary from LSN 0/04016090 or later'
#   at C:/Workspace/code/postgresql/src/bin/pg_walsummary/t/003_tli_switch.pl
line 109.
# Looks like you failed 2 tests of 3.

(test program exited with status code 2)
==============================================================================

The failure log of node-3 is attached. I'll investigate this further
and share my findings but in the meantime if you have any inputs based
on the LOG then do let me know.

-- 
With Regards,
Amit Kapila.

Re: walsummarizer can get stuck when switching timelines

On Tue, Jul 14, 2026 at 6:35 PM Andrey Borodin  wrote:
>
> > On 14 Jul 2026, at 17:49, Amit Kapila  wrote:
> >
> > I tried the patch on Windows and got following failure:
>
> Hi Amit,
>
> Thanks for testing! That output is exactly what an unpatched summarizer produces
> on my machine. I suspect the fix didn't make it into your build.
> Also the test passes for me on Windows CI with the patch applied [0].
>
> Could you double-check that you observe the failure?
>

Yes. This time I tried adding some basic log in walsummarizer.c to
ensure that the test picks up new code. See
extra_debuglog_walsummary_1.txt patch atop the main patch. You can see
the node-3 LOG file (003_tli_switch_node3) attached which has this
following new LOG:
2026-07-15 10:46:56.439 IST walsummarizer[14680] DEBUG:  modified WAL
summarizer started
2026-07-15 10:46:56.440 IST walsummarizer[14680] LOG:  inside
summarizer_wal_segment_open

Is this a sufficient proof or do you think there could be something
else I am missing or messing up?

It passed on my mac but the test took a bit longer than other tests
and the additional observation that the Windows CI passed raises a
question whether this could be some sort of timing issue in the test.
Test timing on mac:
t/001_basic.pl ....... ok      349 ms ( 0.00 usr  0.00 sys +  0.04
cusr  0.03 csys =  0.07 CPU)
t/002_blocks.pl ...... ok     1464 ms ( 0.00 usr  0.00 sys +  0.09
cusr  0.31 csys =  0.40 CPU)
t/003_tli_switch.pl .. ok    13117 ms ( 0.00 usr  0.00 sys +  0.64
cusr  1.85 csys =  2.49 CPU)

Can we first rule out that this can't be a timing issue in the test case itself?

-- 
With Regards,
Amit Kapila.

Re: walsummarizer can get stuck when switching timelines

RE: walsummarizer can get stuck when switching timelines

On Wednesday, July 15, 2026 10:16 PM Robert Haas  wrote:
> On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin  team.ru> wrote:
> > I can reproduce it locally by adding an extra WAL switch on node1
> > before promoting node2. It can probably be made deterministic either
> > by polling pg_stat_archiver before relying on the segment (as
> > 025_stuck_on_old_timeline.pl does) or by setting archive_mode=always
> > on
> > node2 (I would expect it to be a bit faster).
> 
> archive_mode=always seems like the correct fix. Here's v2.

While reading the WAL summarization code, I ran the new 003_tli_switch test and
noticed it takes about 14 seconds to finish on my machine, which prompted me to
debug it a bit. The delay comes from the 10-second wait in the error recovery
path (WAIT_EVENT_WAL_SUMMARIZER_ERROR) in WalSummarizerMain, where it retries
after failing to read an unarchived WAL file.

I was wondering whether it's worth improving this. We could enable summarize_wal
on node3 only after the WALs on the new timeline become readable on node3, which
would eliminate the retry and reduce the test time by about 10 seconds on my
machine. However, if the intention is to exercise the retry logic in
WalSummarizerMain as well, the current test is fine as is. Just sharing this in
case the test time wasn't intentional.

Apart from the test, the fix looks good to me.

Best Regards,
Hou zj