PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Hi PostgreSQL team,
We encountered a standby startup PANIC after a switchover on PostgreSQL 18.6. A standalone reproducer and recovery TAP test reproduce it on 18.6, 17.11, REL_18_STABLE, and master; the same test passes on 18.4 and 17.10.
Expected: a standby that has replayed a heap/visibility-map truncation restarts recovery successfully.
Actual on restart:
LOG: redo starts at 0/3059738 WARNING: page 0 of relation base/5/16396_vm does not exist PANIC: WAL contains references to invalid pages LOG: startup process (PID 55286) was terminated by signal 6: Abort trap: 6
Reproduction: Build PostgreSQL with --enable-tap-tests --enable-injection-points and install injection_points. Copy the attached t/058_vm_truncate_invalid_pages.pl into src/test/recovery/t/, then run:
make -C src/test/recovery check PROVE_TESTS=t/058_vm_truncate_invalid_pages.pl
The test holds a standby restartpoint, promotes that standby, deletes and vacuums an all-visible table (truncating its heap and VM forks), rejoins the old primary as a standby, and restarts it. On 18.6, 20/20 TAP runs failed with this PANIC; on 18.4, 20/20 passed. A shell reproducer, exact steps, WAL excerpts, version matrix, and two candidate patches are included in the attachment.
The affected 18.6 build is REL_18_6 (724edf9); 17.11 is REL_17_11 (083ac03). The runs were on macOS 15 / Darwin 24.6.0, arm64, clang 21.1.8, with debug and injection points enabled and without cassert. The original symptom was on Linux in a three-node streaming setup. The analysis in REPORT.md points to VM block reads without an FPI during redo after an already replayed VM truncation; that diagnosis and the candidate fixes are provided for review.
Regards,
Jacky Nguyen
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
On Sun, 27 Sept 2026 at 11:08, Jacky Nguyen wrote: > > Hi PostgreSQL team, > > We encountered a standby startup PANIC after a switchover on PostgreSQL 18.6. A standalone reproducer and recovery TAP test reproduce it on 18.6, 17.11, REL_18_STABLE, and master; the same test passes on 18.4 and 17.10. > > Expected: a standby that has replayed a heap/visibility-map truncation restarts recovery successfully. > Actual on restart: > > LOG: redo starts at 0/3059738 > WARNING: page 0 of relation base/5/16396_vm does not exist > PANIC: WAL contains references to invalid pages > LOG: startup process (PID 55286) was terminated by signal 6: Abort trap: 6 > > Reproduction: Build PostgreSQL with --enable-tap-tests --enable-injection-points and install injection_points. Copy the attached t/058_vm_truncate_invalid_pages.pl into src/test/recovery/t/, then run: > > make -C src/test/recovery check PROVE_TESTS=t/058_vm_truncate_invalid_pages.pl > > The test holds a standby restartpoint, promotes that standby, deletes and vacuums an all-visible table (truncating its heap and VM forks), rejoins the old primary as a standby, and restarts it. On 18.6, 20/20 TAP runs failed with this PANIC; on 18.4, 20/20 passed. A shell reproducer, exact steps, WAL excerpts, version matrix, and two candidate patches are included in the attachment. > > The affected 18.6 build is REL_18_6 (724edf9); 17.11 is REL_17_11 (083ac03). The runs were on macOS 15 / Darwin 24.6.0, arm64, clang 21.1.8, with debug and injection points enabled and without cassert. The original symptom was on Linux in a three-node streaming setup. The analysis in REPORT.md points to VM block reads without an FPI during redo after an already replayed VM truncation; that diagnosis and the candidate fixes are provided for review. > > Regards, > Jacky Nguyen Hi! Thanks for report. Looks like this bug existed since first VM commit, which was already missing invalid page guard from [0]. So, case from [0] reintroduces one we started to register VM pages. Also I prefer candidate fix b from your email [0] https://github.com/postgres/postgres/commit/defe93463c69f8e0bb717294a34d67c34ac0b03f -- Best regards, Kirill Reshke
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
candidate (a), and it fixes this report too. On master the attached test
fails without it and passes with it. The same change also passes on 17,
18 and 19.
This does not need a switchover. With full_page_writes = off, a plain
standby restart hits it if the restartpoint is before a DELETE or UPDATE
that cleared VM bits and a later VACUUM truncated the VM. So it would be
good to get v2-0002 in before the next minor release.
I asked Fable to create a test for me.
The attached clears VM bits by delete, HOT update
and cross-page update, and fails if any one of the three reads still uses
RBM_NORMAL.
Candidate (b) would also work, but once the read no longer uses
RBM_NORMAL, no invalid page is logged for the VM, so it is not needed.
Until the fix ships, ignore_invalid_pages = on lets the standby start. It
can be turned off after the next restartpoint.
[1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com
Thanks,
Shihao
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Hi, On Mon, 28 Sep 2026, shihao zhong wrote: > Melanie's v2-0002 in the VM clear thread [1] is the same change as > candidate (a), and it fixes this report too. I tested Melanie's v2 series as posted (0001-0003, which apply to REL_19_STABLE) and candidate (a) on REL_18_STABLE, using the steps from Shihao's test: full_page_writes = off and a plain standby restart. Both were debug builds with assertions, and the fix works: - Unpatched, 18 and 19 fail to restart with "WAL contains references to invalid pages" after "page 0 of relation ..._vm does not exist" for each table. - Patched, the standby restarts and its VM forks end up truncated again. - On 18, reverting any one of the three RBM_ZERO_ON_ERROR changes brings the PANIC back, so the test covers each of them. However, with wal_consistency_checking = all on the primary, the patched standby still fails to restart: FATAL: invalid page pd_lower 0 pd_upper 0 pd_special 0 CONTEXT: WAL redo at 0/03A80068 for Heap/DELETE: ...; blkref #1: rel 1663/5/16384, fork 2, blk 0 FPW RBM_ZERO_ON_ERROR recreates the truncated VM page as all zeros. verifyBackupPageConsistency() then masks it with heap_mask(), and mask_unused_space() rejects a page with pd_lower 0. heap_xlog_prune_freeze() and heap_xlog_multi_insert() already initialize a VM page that was read as zeros, and doing the same at the three VM clear sites fixes it. The attached diff applies on top of v2 on REL_19_STABLE; with it, the standby restarts with and without wal_consistency_checking. On 18, candidate (b) alone also passes with wal_consistency_checking, since it never recreates the page. This only matters with wal_consistency_checking, but buildfarm animals that use it could trip over Shihao's test once it's in. Also, v2-0002 applies only on top of v2-0001 on REL_19_STABLE. On master it doesn't apply after master-v2-0001, 18 needs its own version (candidate (a) applies there as is), and in 17 the redo code is in heapam.c. [1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com Regards, Rahul Yadav
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
On Thu, Oct 1, 2026, rahul@rhyadav.dev wrote:
> However, with wal_consistency_checking = all on the primary, the
> patched standby still fails to restart:
I can reproduce this on master, and your change fixes it. It matches
heap_xlog_prune_freeze(), and vm_readbuf() on the primary.
No posted version of the fix applies to master on its own, so here
is one as v3. I will add a CommitFest entry so this does not get lost
before the November releases.
0001 is Melanie's v2-0002 from [1]. The changed lines and the commit
message are hers, only the context is adjusted for master. I left out
her v2-0001 and v2-0003, this report does not need them.
0002 is your diff. It had no commit message, so I put one together
from your mail. Please correct it if it is wrong.
0003 is my test. It now sets wal_consistency_checking = all, and it
takes a row lock first. Without the lock, the VM_NEW read in
heap_xlog_update() is not covered once 0002 is in.
The test fails without 0001. It also fails if any one of the three
RBM_ZERO_ON_ERROR or the three PageInit() calls is removed. That
holds on master, 19, 18 and 17. The nocfbot files hold the same three
patches for the back branches.
Melanie, if you would rather handle this in your thread, tell me and
I will close the entry.
[1] https://postgr.es/m/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0%3D%2BidehHw%40mail.gmail.com
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Hi Shihao, On Fri, 2 Oct 2026, shihao zhong wrote: > 0002 is your diff. It had no commit message, so I put one together > from your mail. Please correct it if it is wrong. Thanks for putting v3 together. The message is right for master and 19. On 17 and 18 the function that already does this is heap_xlog_visible(), so in those versions the sentence would be: heap_xlog_visible() already initializes a VM page that was read as zeros. 0002 could also carry the same "Backpatch-through: 17" as 0001. I ran my reproducer, which follows the same steps as 0003, against v3 on master and the nocfbot versions on 19, 18 and 17. Unpatched master and 17 fail to restart with the invalid pages PANIC. With v3 the standby restarts cleanly on all four branches, in each of these setups: - full_page_writes = off - full_page_writes = off, wal_consistency_checking = all - full_page_writes = on, wal_consistency_checking = all 0003 itself passes here too on all four branches, and fails on master without 0001 and 0002. Regards, Rahul Yadav
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Thanks for testing v3 on all four branches.
You are right about 17 and 18. I will change that sentence to
heap_xlog_visible() in those two versions, and add
"Backpatch-through: 17" to 0002. Both are commit message changes
only, so I will fold them into the next version and not post a v4
just for that.
I added a CommitFest entry so this is tracked for the November
releases.
Shihao
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Hi Shihao, cfbot says v3 needs a rebase. 0001 and 0002 still apply to master, but the test's number is now taken: 12f866cd5f and cc33773482 added 058_shutdown_crash_restart.pl and 059_remote_apply_status_interval.pl to src/test/recovery. With the test renamed to 060_vm_clear_truncate.pl, the series applies to master (16b94c4012) and the test passes. As you were going to fold the heap_xlog_visible() wording for 17 and 18 into the next version, this may be a good time for v4. Regards, Rahul Yadav Oct 3, 2026, 05:13 by zhong950419@gmail.com: > Hi Rahul, > > Thanks for testing v3 on all four branches. > > You are right about 17 and 18. I will change that sentence to > heap_xlog_visible() in those two versions, and add > "Backpatch-through: 17" to 0002. Both are commit message changes > only, so I will fold them into the next version and not post a v4 > just for that. > > I added a CommitFest entry so this is tracked for the November > releases. > > https://commitfest.postgresql.org/patch/7386/ > > Thanks, > Shihao >
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Thanks for checking. Here is v4, rebased on the current master.
The test is now 060_vm_clear_truncate.pl, on the back branches too.
No code changes.
0002 now has "Backpatch-through: 17". In the 17 and 18 files its
commit message names heap_xlog_visible(), as you pointed out.
Shihao
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Jacky's original fix with XLogTruncateRelation() is the one I prefer. I'm leaning toward sticking with RBM_NORMAL for heap_xlog_vm_clear() as I don't see it halting recovery except for the truncation case, so the natural solution to the truncation issue is to do what we do for the main fork. On Thu, Oct 8, 2026 at 11:13 PM shihao zhong wrote: > > The test is now 060_vm_clear_truncate.pl, on the back branches too. > No code changes. Attached patch uses pg_visibility to truncate and then piggy-backs on one of its existing tests. - Melanie
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
> Jacky's original fix with XLogTruncateRelation() is the one I prefer.
standby restarts.
One small thing. On 17 the test part does not apply, because
001_concurrent_transaction.pl has no standby there.
Thanks,
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
On Tue, Oct 13, 2026 at 11:20 PM shihao zhong wrote: > > > Jacky's original fix with XLogTruncateRelation() is the one I prefer. > > Fine with me. I tested v5 on master, 19, 18 and 17. With the fix the > standby restarts. While working on committing this, an AI review pointed out that the HEAPBLK_TO_MAPBLOCK_LIMIT() macro is not overflow safe and needs to be fixed. This patch adds a user of visibilitymap_truncation_length(), but the existing user, wal summarizer can already encounter this bug. A heap block number near MaxBlockNumber can be passed it and then adding HEAPBLOCKS_PER_PAGE - 1 overflows it and wraps it to a small number, recording a VM limit much smaller than is correct and causing the summarizer to discard tracked VM modifications for the rest of the fork. I think the attached preliminary patch 0001 has to be committed before 0002. We don't have a good way of adding a regression test for this that doesn't take forever because of the large heap block number required. > One small thing. On 17 the test part does not apply, because > 001_concurrent_transaction.pl has no standby there. Thanks for the heads up. I've implemented the 17 port locally. - Melanie
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
Thanks for catching this. 0001 looks right to me, and committing it
first makes sense.
Re: PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation
On Mon, Nov 16, 2026 at 12:10 PM shihao zhong wrote: > > Just curious, I think the overflow could only happen in 32bit build, do > we still support it? You can see 32-bit animals on the buildfarm. Also, I think it's better not to have this macro construct an intermediate value larger than the input value. - Melanie