PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

Поиск
Список
Период
Сортировка
Искать

PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Jacky Nguyen <nktpro@gmail.com>
Дата:
Вложения:

Hi PostgreSQL team,

We encountered a standby startup PANIC after a switchover on PostgreSQL 18.6. A standalone reproducer and recovery TAP test reproduce it on 18.6, 17.11, REL_18_STABLE, and master; the same test passes on 18.4 and 17.10.

Expected: a standby that has replayed a heap/visibility-map truncation restarts recovery successfully.
Actual on restart:

LOG:  redo starts at 0/3059738
WARNING:  page 0 of relation base/5/16396_vm does not exist
PANIC:  WAL contains references to invalid pages
LOG:  startup process (PID 55286) was terminated by signal 6: Abort trap: 6

Reproduction: Build PostgreSQL with --enable-tap-tests --enable-injection-points and install injection_points. Copy the attached t/058_vm_truncate_invalid_pages.pl into src/test/recovery/t/, then run:

make -C src/test/recovery check PROVE_TESTS=t/058_vm_truncate_invalid_pages.pl

The test holds a standby restartpoint, promotes that standby, deletes and vacuums an all-visible table (truncating its heap and VM forks), rejoins the old primary as a standby, and restarts it. On 18.6, 20/20 TAP runs failed with this PANIC; on 18.4, 20/20 passed. A shell reproducer, exact steps, WAL excerpts, version matrix, and two candidate patches are included in the attachment.

The affected 18.6 build is REL_18_6 (724edf9); 17.11 is REL_17_11 (083ac03). The runs were on macOS 15 / Darwin 24.6.0, arm64, clang 21.1.8, with debug and injection points enabled and without cassert. The original symptom was on Linux in a three-node streaming setup. The analysis in REPORT.md points to VM block reads without an FPI during redo after an already replayed VM truncation; that diagnosis and the candidate fixes are provided for review.

Regards,
Jacky Nguyen

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Kirill Reshke <reshkekirill@gmail.com>
Дата:
On Sun, 27 Sept 2026 at 11:08, Jacky Nguyen  wrote:
>
> Hi PostgreSQL team,
>
> We encountered a standby startup PANIC after a switchover on PostgreSQL 18.6. A standalone reproducer and recovery TAP test reproduce it on 18.6, 17.11, REL_18_STABLE, and master; the same test passes on 18.4 and 17.10.
>
> Expected: a standby that has replayed a heap/visibility-map truncation restarts recovery successfully.
> Actual on restart:
>
> LOG:  redo starts at 0/3059738
> WARNING:  page 0 of relation base/5/16396_vm does not exist
> PANIC:  WAL contains references to invalid pages
> LOG:  startup process (PID 55286) was terminated by signal 6: Abort trap: 6
>
> Reproduction: Build PostgreSQL with --enable-tap-tests --enable-injection-points and install injection_points. Copy the attached t/058_vm_truncate_invalid_pages.pl into src/test/recovery/t/, then run:
>
> make -C src/test/recovery check PROVE_TESTS=t/058_vm_truncate_invalid_pages.pl
>
> The test holds a standby restartpoint, promotes that standby, deletes and vacuums an all-visible table (truncating its heap and VM forks), rejoins the old primary as a standby, and restarts it. On 18.6, 20/20 TAP runs failed with this PANIC; on 18.4, 20/20 passed. A shell reproducer, exact steps, WAL excerpts, version matrix, and two candidate patches are included in the attachment.
>
> The affected 18.6 build is REL_18_6 (724edf9); 17.11 is REL_17_11 (083ac03). The runs were on macOS 15 / Darwin 24.6.0, arm64, clang 21.1.8, with debug and injection points enabled and without cassert. The original symptom was on Linux in a three-node streaming setup. The analysis in REPORT.md points to VM block reads without an FPI during redo after an already replayed VM truncation; that diagnosis and the candidate fixes are provided for review.
>
> Regards,
> Jacky Nguyen

Hi!
Thanks for report. Looks like this bug existed since first VM commit,
which was already missing invalid page guard from [0].
So, case from [0] reintroduces one we started to register VM pages.

Also I prefer candidate fix b from your email

[0] https://github.com/postgres/postgres/commit/defe93463c69f8e0bb717294a34d67c34ac0b03f


-- 
Best regards,
Kirill Reshke


Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Вложения:
Hi Jacky and Kirill,

Melanie's v2-0002 in the VM clear thread [1] is the same change as
candidate (a), and it fixes this report too. On master the attached test
fails without it and passes with it. The same change also passes on 17,
18 and 19.

This does not need a switchover. With full_page_writes = off, a plain
standby restart hits it if the restartpoint is before a DELETE or UPDATE
that cleared VM bits and a later VACUUM truncated the VM. So it would be
good to get v2-0002 in before the next minor release.

I asked Fable to create a test for me. 

The attached clears VM bits by delete, HOT update
and cross-page update, and fails if any one of the three reads still uses
RBM_NORMAL.

Candidate (b) would also work, but once the read no longer uses
RBM_NORMAL, no invalid page is logged for the VM, so it is not needed.

Until the fix ships, ignore_invalid_pages = on lets the standby start. It
can be turned off after the next restartpoint.

[1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
rahul@rhyadav.dev
Дата:
Вложения:
Hi,

On Mon, 28 Sep 2026, shihao zhong wrote:
> Melanie's v2-0002 in the VM clear thread [1] is the same change as
> candidate (a), and it fixes this report too.

I tested Melanie's v2 series as posted (0001-0003, which apply to
REL_19_STABLE) and candidate (a) on REL_18_STABLE, using the steps
from Shihao's test: full_page_writes = off and a plain standby
restart.  Both were debug builds with assertions, and the fix works:

- Unpatched, 18 and 19 fail to restart with "WAL contains references
  to invalid pages" after "page 0 of relation ..._vm does not exist"
  for each table.
- Patched, the standby restarts and its VM forks end up truncated
  again.
- On 18, reverting any one of the three RBM_ZERO_ON_ERROR changes
  brings the PANIC back, so the test covers each of them.

However, with wal_consistency_checking = all on the primary, the
patched standby still fails to restart:

  FATAL:  invalid page pd_lower 0 pd_upper 0 pd_special 0
  CONTEXT:  WAL redo at 0/03A80068 for Heap/DELETE: ...;
  blkref #1: rel 1663/5/16384, fork 2, blk 0 FPW

RBM_ZERO_ON_ERROR recreates the truncated VM page as all zeros.
verifyBackupPageConsistency() then masks it with heap_mask(), and
mask_unused_space() rejects a page with pd_lower 0.
heap_xlog_prune_freeze() and heap_xlog_multi_insert() already
initialize a VM page that was read as zeros, and doing the same at
the three VM clear sites fixes it.  The attached diff applies on top
of v2 on REL_19_STABLE; with it, the standby restarts with and
without wal_consistency_checking.  On 18, candidate (b) alone also
passes with wal_consistency_checking, since it never recreates the
page.

This only matters with wal_consistency_checking, but buildfarm
animals that use it could trip over Shihao's test once it's in.

Also, v2-0002 applies only on top of v2-0001 on REL_19_STABLE.  On
master it doesn't apply after master-v2-0001, 18 needs its own
version (candidate (a) applies there as is), and in 17 the redo code
is in heapam.c.

[1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com

Regards,
Rahul Yadav


Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Вложения:
Hi Rahul,

On Thu, Oct 1, 2026, rahul@rhyadav.dev wrote:
> However, with wal_consistency_checking = all on the primary, the
> patched standby still fails to restart:

I can reproduce this on master, and your change fixes it. It matches
heap_xlog_prune_freeze(), and vm_readbuf() on the primary.

No posted version of the fix applies to master on its own, so here
is one as v3. I will add a CommitFest entry so this does not get lost
before the November releases.

0001 is Melanie's v2-0002 from [1]. The changed lines and the commit
message are hers, only the context is adjusted for master. I left out
her v2-0001 and v2-0003, this report does not need them.

0002 is your diff. It had no commit message, so I put one together
from your mail. Please correct it if it is wrong.

0003 is my test. It now sets wal_consistency_checking = all, and it
takes a row lock first. Without the lock, the VM_NEW read in
heap_xlog_update() is not covered once 0002 is in.

The test fails without 0001. It also fails if any one of the three
RBM_ZERO_ON_ERROR or the three PageInit() calls is removed. That
holds on master, 19, 18 and 17. The nocfbot files hold the same three
patches for the back branches.

Melanie, if you would rather handle this in your thread, tell me and
I will close the entry.

[1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
rahul@rhyadav.dev
Дата:
Hi Shihao,

On Fri, 2 Oct 2026, shihao zhong wrote:
> 0002 is your diff. It had no commit message, so I put one together
> from your mail. Please correct it if it is wrong.

Thanks for putting v3 together.  The message is right for master and
19.  On 17 and 18 the function that already does this is
heap_xlog_visible(), so in those versions the sentence would be:

  heap_xlog_visible() already initializes a VM page that was read as
  zeros.

0002 could also carry the same "Backpatch-through: 17" as 0001.

I ran my reproducer, which follows the same steps as 0003, against v3
on master and the nocfbot versions on 19, 18 and 17.  Unpatched
master and 17 fail to restart with the invalid pages PANIC.  With v3
the standby restarts cleanly on all four branches, in each of these
setups:

  - full_page_writes = off
  - full_page_writes = off, wal_consistency_checking = all
  - full_page_writes = on, wal_consistency_checking = all

0003 itself passes here too on all four branches, and fails on
master without 0001 and 0002.

Regards,
Rahul Yadav




Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Hi Rahul,

Thanks for testing v3 on all four branches.

You are right about 17 and 18. I will change that sentence to
heap_xlog_visible() in those two versions, and add
"Backpatch-through: 17" to 0002. Both are commit message changes
only, so I will fold them into the next version and not post a v4
just for that.

I added a CommitFest entry so this is tracked for the November
releases.

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Rahul <rahul@rhyadav.dev>
Дата:
Hi Shihao,

cfbot says v3 needs a rebase.  0001 and 0002 still apply to master,
but the test's number is now taken: 12f866cd5f and cc33773482 added
058_shutdown_crash_restart.pl and 059_remote_apply_status_interval.pl
to src/test/recovery.  With the test renamed to
060_vm_clear_truncate.pl, the series applies to master (16b94c4012)
and the test passes.

As you were going to fold the heap_xlog_visible() wording for 17 and
18 into the next version, this may be a good time for v4.

Regards,
Rahul Yadav


Oct 3, 2026, 05:13 by zhong950419@gmail.com:

> Hi Rahul,
>
> Thanks for testing v3 on all four branches.
>
> You are right about 17 and 18. I will change that sentence to
> heap_xlog_visible() in those two versions, and add
> "Backpatch-through: 17" to 0002. Both are commit message changes
> only, so I will fold them into the next version and not post a v4
> just for that.
>
> I added a CommitFest entry so this is tracked for the November
> releases.
>
> https://commitfest.postgresql.org/patch/7386/
>
> Thanks,
> Shihao
>



Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Вложения:
Hi Rahul,

Thanks for checking. Here is v4, rebased on the current master.

The test is now 060_vm_clear_truncate.pl, on the back branches too.
No code changes.

0002 now has "Backpatch-through: 17". In the 17 and 18 files its
commit message names heap_xlog_visible(), as you pointed out.

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Melanie Plageman <melanieplageman@gmail.com>
Дата:
Вложения:
Jacky's original fix with XLogTruncateRelation() is the one I prefer.
I'm leaning toward sticking with RBM_NORMAL for heap_xlog_vm_clear()
as I don't see it halting recovery except for the truncation case, so
the natural solution to the truncation issue is to do what we do for
the main fork.

On Thu, Oct 8, 2026 at 11:13 PM shihao zhong  wrote:
>
> The test is now 060_vm_clear_truncate.pl, on the back branches too.
> No code changes.

Attached patch uses pg_visibility to truncate and then piggy-backs on
one of its existing tests.

- Melanie

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Hi Melanie,

> Jacky's original fix with XLogTruncateRelation() is the one I prefer.

Fine with me. I tested v5 on master, 19, 18 and 17. With the fix the
standby restarts.

One small thing. On 17 the test part does not apply, because
001_concurrent_transaction.pl has no standby there.

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Melanie Plageman <melanieplageman@gmail.com>
Дата:
Вложения:
On Tue, Oct 13, 2026 at 11:20 PM shihao zhong  wrote:
>
> > Jacky's original fix with XLogTruncateRelation() is the one I prefer.
>
> Fine with me. I tested v5 on master, 19, 18 and 17. With the fix the
> standby restarts.

While working on committing this, an AI review pointed out that the
HEAPBLK_TO_MAPBLOCK_LIMIT() macro is not overflow safe and needs to be
fixed. This patch adds a user of visibilitymap_truncation_length(),
but the existing user, wal summarizer can already encounter this bug.
A heap block number near MaxBlockNumber can be passed it and then
adding HEAPBLOCKS_PER_PAGE - 1 overflows it and wraps it to a small
number, recording a VM limit much smaller than is correct and causing
the summarizer to discard tracked VM modifications for the rest of the
fork.

I think the attached preliminary patch 0001 has to be committed before 0002.

We don't have a good way of adding a regression test for this that
doesn't take forever because of the large heap block number required.

> One small thing. On 17 the test part does not apply, because
> 001_concurrent_transaction.pl has no standby there.

Thanks for the heads up. I've implemented the 17 port locally.

- Melanie

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Hi Melanie,

Thanks for catching this. 0001 looks right to me, and committing it
first makes sense.

Just curious, I think the overflow could only happen in 32bit build, do
we still support it?

Thanks,
Shihao

Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
Melanie Plageman <melanieplageman@gmail.com>
Дата:
On Mon, Nov 16, 2026 at 12:10 PM shihao zhong  wrote:
>
> Just curious, I think the overflow could only happen in 32bit build, do
> we still support it?

You can see 32-bit animals on the buildfarm. Also, I think it's better
not to have this macro construct an intermediate value larger than the
input value.

- Melanie


Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Hi Melanie,

Thanks for explaining, I appreciate it!

Thanks,
Shihao
FAQ