BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
От:
PG Bug reporting form <noreply@postgresql.org>
Дата:
The following bug has been logged on the website:
Bug reference: 19839
Logged by: Atish Prakash
Email address: 125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system: Linux x86_64 (kernel 7.0.0, glibc 2.43)
Description:
On an assert-enabled build, logical decoding can fail this assertion and
take
the server through crash recovery:
TRAP: failed Assert("XLogRecPtrIsValid(txn->final_lsn)"), File:
"reorderbuffer.c", Line: 5001
Expected: decoding discards the aborted transaction and continues.
The sequence is:
1. A transaction has a subtransaction with an XID but no decoded changes,
for
example one that only ran SELECT ... FOR UPDATE.
2. The parent exceeds logical_decoding_work_mem while still in progress.
ReorderBufferSerializeTXN() recurses into the subtransaction, writes
nothing,
and still sets RBTXN_IS_SERIALIZED on it. Its final_lsn stays invalid.
3. The parent aborts while the decoder is behind.
4. The parent is chosen for eviction before its abort record is decoded.
ReorderBufferCheckAndTruncateAbortedTXN() finds it aborted and truncates
it,
and ReorderBufferRestoreCleanup() fails on the subtransaction's
final_lsn.
ReorderBufferCheckAndTruncateAbortedTXN() came in with 072ee847ad, so I
expect
17 and earlier are not affected, but I have not tested them.
From reading the code, a non-assert build goes through an empty cleanup loop
for
this subtransaction, so I expect no user-visible effect there. I have not
run a
non-assert build.
I found this with a stress test of REPACK (CONCURRENTLY), where the REPACK
decoding worker failed in the same way, on both a plain assert build and an
ASan
build. It does not need REPACK: the attached script uses only
pg_recvlogical. It
stalls pg_recvlogical with SIGSTOP so that the decoder is behind when the
transaction aborts.
Proposed fix, attached as 0001-fix-aborted-spilled-txn-assert.patch: set
RBTXN_IS_SERIALIZED only if something was written. With the patch, on
master:
- the script no longer fails
- all test_decoding tests and all 239 core regression tests pass
- a spilled transaction with an empty subtransaction, a rolled-back
subtransaction
and a released one decodes the expected rows
TAP tests were not run, because my build has them disabled. The patch has no
regression test, because the script depends on stalling a live consumer. It
applies cleanly to REL_18_STABLE and REL_19_STABLE, but I only built and
tested it
on master.
I found this with a fuzzer, and AI tools helped me build it, write the
script
and patch, and draft this report.
Versions where the script fails:
- 18.6 (REL_18_STABLE at 813a74bc5a; the assertion reads txn->final_lsn !=
InvalidXLogRecPtr)
- 19rc1 (REL_19_STABLE at 0b1eb8b917)
- master at 72b9498df9, plus my unrelated uncommitted patch to ruleutils.c
The patch was built on master at b58a9a97fe.
Platform: Linux 7.0.0-42-generic x86_64, glibc 2.43, Intel i9-13900HX, 14
GiB RAM
Build: clang 21.1.8, CFLAGS=-O1 -g, --enable-cassert --enable-debug
--without-icu --without-readline --without-libxml --without-openssl
Settings set by the script: wal_level=logical,
logical_decoding_work_mem=64kB
Stack from 19rc1, with addr2line for the static functions:
ReorderBufferRestoreCleanup reorderbuffer.c:5001
ReorderBufferTruncateTXN reorderbuffer.c:1749
(subtransaction)
ReorderBufferTruncateTXN reorderbuffer.c:1665 (parent)
ReorderBufferCheckAndTruncateAbortedTXN reorderbuffer.c:1819
ReorderBufferQueueChange
DecodeInsert decode.c:956
LogicalDecodingProcessRecord
Reproducer, run as: logical-decoding-assert-repro.sh
#!/bin/bash
# Usage: logical-decoding-assert-repro.sh [work dir]
# Assertion failure in ReorderBufferRestoreCleanup during logical decoding,
without REPACK.
set -u
BIN=$1; WORK=${2:-$(mktemp -d)}; PORT=${PORT:-6399}; D=$WORK/data; rm -rf
"$D"; mkdir -p "$WORK"
export PGHOST=$WORK PGPORT=$PORT PGDATABASE=postgres
"$BIN/initdb" -D "$D" -A trust --no-sync >/dev/null
cat >> "$D/postgresql.conf" </dev/null
P="$BIN/psql -X -q -v ON_ERROR_STOP=1"
$P -c "CREATE TABLE t (a text)" -c "INSERT INTO t VALUES ('locked')" \
-c "SELECT pg_create_logical_replication_slot('s', 'test_decoding')"
>/dev/null
"$BIN/pg_recvlogical" -d postgres -S s --start -f /dev/null -F 0 -s 1 &
RECV=$!
# Session A stays open throughout.
FIFO=$WORK/a.fifo; rm -f "$FIFO"; mkfifo "$FIFO"; $P < "$FIFO" > /dev/null &
APSQL=$!; exec 9> "$FIFO"
# A: a subtransaction with an XID but no decoded change, then enough to
exceed the limit.
cat >&9 <&9
sleep 0.5
$P <&9
exec 9>&-
wait $APSQL
kill -CONT $RECV # decoding resumes; A's abort record has not been read
but CLOG says aborted
sleep 8
wait $DS 2>/dev/null
kill $RECV 2>/dev/null
echo "--- server log"
grep -E 'TRAP|terminated by signal|was terminated' "$WORK/server.log" | cut
-c1-200
"$BIN/pg_ctl" -D "$D" -m immediate stop >/dev/null 2>&1
Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
On Sat, 7 Nov 2026 at 06:54, PG Bug reporting form <noreply@postgresql.org> wrote:
The following bug has been logged on the website:
Bug reference: 19839
Logged by: Atish Prakash
Email address: 125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system: Linux x86_64 (kernel 7.0.0, glibc 2.43)
Description:
On an assert-enabled build, logical decoding can fail this assertion and
take
the server through crash recovery:
TRAP: failed Assert("XLogRecPtrIsValid(txn->final_lsn)"), File:
"reorderbuffer.c", Line: 5001
Expected: decoding discards the aborted transaction and continues.
The sequence is:
1. A transaction has a subtransaction with an XID but no decoded changes,
for
example one that only ran SELECT ... FOR UPDATE.
2. The parent exceeds logical_decoding_work_mem while still in progress.
ReorderBufferSerializeTXN() recurses into the subtransaction, writes
nothing,
and still sets RBTXN_IS_SERIALIZED on it. Its final_lsn stays invalid.
3. The parent aborts while the decoder is behind.
4. The parent is chosen for eviction before its abort record is decoded.
ReorderBufferCheckAndTruncateAbortedTXN() finds it aborted and truncates
it,
and ReorderBufferRestoreCleanup() fails on the subtransaction's
final_lsn.
ReorderBufferCheckAndTruncateAbortedTXN() came in with 072ee847ad, so I
expect
17 and earlier are not affected, but I have not tested them.
From reading the code, a non-assert build goes through an empty cleanup loop
for
this subtransaction, so I expect no user-visible effect there. I have not
run a
non-assert build.
I found this with a stress test of REPACK (CONCURRENTLY), where the REPACK
decoding worker failed in the same way, on both a plain assert build and an
ASan
build. It does not need REPACK: the attached script uses only
pg_recvlogical. It
stalls pg_recvlogical with SIGSTOP so that the decoder is behind when the
transaction aborts.
Proposed fix, attached as 0001-fix-aborted-spilled-txn-assert.patch: set
RBTXN_IS_SERIALIZED only if something was written. With the patch, on
master:
- the script no longer fails
- all test_decoding tests and all 239 core regression tests pass
- a spilled transaction with an empty subtransaction, a rolled-back
subtransaction
and a released one decodes the expected rows
TAP tests were not run, because my build has them disabled. The patch has no
regression test, because the script depends on stalling a live consumer. It
applies cleanly to REL_18_STABLE and REL_19_STABLE, but I only built and
tested it
on master.
I found this with a fuzzer, and AI tools helped me build it, write the
script
and patch, and draft this report.
Versions where the script fails:
- 18.6 (REL_18_STABLE at 813a74bc5a; the assertion reads txn->final_lsn !=
InvalidXLogRecPtr)
- 19rc1 (REL_19_STABLE at 0b1eb8b917)
- master at 72b9498df9, plus my unrelated uncommitted patch to ruleutils.c
The patch was built on master at b58a9a97fe.
Platform: Linux 7.0.0-42-generic x86_64, glibc 2.43, Intel i9-13900HX, 14
GiB RAM
Build: clang 21.1.8, CFLAGS=-O1 -g, --enable-cassert --enable-debug
--without-icu --without-readline --without-libxml --without-openssl
Settings set by the script: wal_level=logical,
logical_decoding_work_mem=64kB
Stack from 19rc1, with addr2line for the static functions:
ReorderBufferRestoreCleanup reorderbuffer.c:5001
ReorderBufferTruncateTXN reorderbuffer.c:1749
(subtransaction)
ReorderBufferTruncateTXN reorderbuffer.c:1665 (parent)
ReorderBufferCheckAndTruncateAbortedTXN reorderbuffer.c:1819
ReorderBufferQueueChange
DecodeInsert decode.c:956
LogicalDecodingProcessRecord
Reproducer, run as: logical-decoding-assert-repro.sh <bin dir of an assert
build>
#!/bin/bash
# Usage: logical-decoding-assert-repro.sh <postgres bin dir, built with
--enable-cassert> [work dir]
# Assertion failure in ReorderBufferRestoreCleanup during logical decoding,
without REPACK.
set -u
BIN=$1; WORK=${2:-$(mktemp -d)}; PORT=${PORT:-6399}; D=$WORK/data; rm -rf
"$D"; mkdir -p "$WORK"
export PGHOST=$WORK PGPORT=$PORT PGDATABASE=postgres
"$BIN/initdb" -D "$D" -A trust --no-sync >/dev/null
cat >> "$D/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$WORK'
port = $PORT
wal_level = logical
logical_decoding_work_mem = 64kB
CONF
"$BIN/pg_ctl" -D "$D" -l "$WORK/server.log" -w start >/dev/null
P="$BIN/psql -X -q -v ON_ERROR_STOP=1"
$P -c "CREATE TABLE t (a text)" -c "INSERT INTO t VALUES ('locked')" \
-c "SELECT pg_create_logical_replication_slot('s', 'test_decoding')"
>/dev/null
"$BIN/pg_recvlogical" -d postgres -S s --start -f /dev/null -F 0 -s 1 &
RECV=$!
# Session A stays open throughout.
FIFO=$WORK/a.fifo; rm -f "$FIFO"; mkfifo "$FIFO"; $P < "$FIFO" > /dev/null &
APSQL=$!; exec 9> "$FIFO"
# A: a subtransaction with an XID but no decoded change, then enough to
exceed the limit.
cat >&9 <<SQL
BEGIN;
INSERT INTO t VALUES ('first');
SAVEPOINT sp;
SELECT * FROM t WHERE a = 'locked' FOR UPDATE;
RELEASE sp;
INSERT INTO t SELECT repeat('a', 100) FROM generate_series(1, 2000);
SELECT 1;
SQL
sleep 2 # the walsender decodes this and spills A; A is still in progress
# Stall the walsender in the middle of sending a large committed
transaction.
kill -STOP $RECV
$P -c "INSERT INTO t SELECT repeat('c', 1000) FROM generate_series(1,
200000)"
for i in $(seq 1 120); do
w=$($P -Atc "SELECT wait_event FROM pg_stat_activity WHERE backend_type =
'walsender'")
[ "$w" = "WalSenderWriteData" ] && break
sleep 0.5
done
echo "walsender wait event: $w"
# Behind the stalled walsender: more of A, then another transaction that
crosses the limit, then A aborts.
echo "INSERT INTO t SELECT repeat('a', 100) FROM generate_series(1, 200);
SELECT 1;" >&9
sleep 0.5
$P <<SQL &
BEGIN;
INSERT INTO t SELECT repeat('d', 100) FROM generate_series(1, 120);
SELECT pg_sleep(4);
COMMIT;
SQL
DS=$!
sleep 1
echo "ROLLBACK;" >&9
exec 9>&-
wait $APSQL
kill -CONT $RECV # decoding resumes; A's abort record has not been read
but CLOG says aborted
sleep 8
wait $DS 2>/dev/null
kill $RECV 2>/dev/null
echo "--- server log"
grep -E 'TRAP|terminated by signal|was terminated' "$WORK/server.log" | cut
-c1-200
"$BIN/pg_ctl" -D "$D" -m immediate stop >/dev/null 2>&1
Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
Hi,
On Sat, Nov 7, 2026 at 09:26 AM Atisss hh <125atishprak@gmail.com>wrote:
>
>
> On Sat, 7 Nov 2026 at 06:54, PG Bug reporting form
> > wrote:
>
> The following bug has been logged on the website:
>
> Bug reference: 19839
> Logged by: Atish Prakash
> Email address: 125atishprak@gmail.com
> PostgreSQL version: 18.6
> Operating system: Linux x86_64 (kernel 7.0.0, glibc 2.43)
> Description:
>
> On an assert-enabled build, logical decoding can fail this assertion and
> take
> the server through crash recovery:
>
> TRAP: failed Assert("XLogRecPtrIsValid(txn->final_lsn)"), File:
> "reorderbuffer.c", Line: 5001
>
> Expected: decoding discards the aborted transaction and continues.
>
> The sequence is:
>
> 1. A transaction has a subtransaction with an XID but no decoded
> changes,
> for
> example one that only ran SELECT ... FOR UPDATE.
> 2. The parent exceeds logical_decoding_work_mem while still in progress.
> ReorderBufferSerializeTXN() recurses into the subtransaction, writes
> nothing,
> and still sets RBTXN_IS_SERIALIZED on it. Its final_lsn stays
> invalid.
> 3. The parent aborts while the decoder is behind.
> 4. The parent is chosen for eviction before its abort record is decoded.
> ReorderBufferCheckAndTruncateAbortedTXN() finds it aborted and
> truncates
> it,
> and ReorderBufferRestoreCleanup() fails on the subtransaction's
> final_lsn.
>
> ReorderBufferCheckAndTruncateAbortedTXN() came in with 072ee847ad, so I
> expect
> 17 and earlier are not affected, but I have not tested them.
Thanks for the report and patch.
I think this issue affects can all supported branches (back to PG 14),
and does not require an aborted transaction to reproduce.
For example, if a top-level transaction has a subtransaction with an XID
but no decoded changes, and then spills to disk while queuing a partial
change (such as TOAST chunks), the empty subtransaction is marked
RBTXN_IS_SERIALIZED with final_lsn == InvalidXLogRecPtr. Once the main
table insert completes the partial change,
ReorderBufferProcessPartialChange() streams the transaction via
ReorderBufferStreamTXN(), and ReorderBufferTruncateTXN() ->
ReorderBufferRestoreCleanup() hits the same assertion on the empty
subtransaction.
See the attachment for a test to reproduce this issue. (Feel free to
merge if it's worth adding a test to cover this path)
>
> From reading the code, a non-assert build goes through an empty
> cleanup loop
> for
> this subtransaction, so I expect no user-visible effect there. I
> have not
> run a
> non-assert build.
I think in a rare case, this can also affect the stats collection for
spilled txns. For example, if an empty subtransaction is marked as
serialized, we skip incrementing spillTxns because no changes were
written; if real changes are later serialized in this subtransaction, we
still do not increment spillTxns because the subtransaction is already
marked as serialized and we avoid counting it "twice".
>
> Proposed fix, attached as 0001-fix-aborted-spilled-txn-assert.patch: set
> RBTXN_IS_SERIALIZED only if something was written.
And I have one minor comment for the patch:
+ if (spilled)
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
ReorderBufferSerializeTXN() already has an "if (spilled)" block right
above this, so we shall directly move the assignment into that block.
--
Best Regards,
Zhijie HouRe: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
От:
Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Дата:
Hi, On Sun, Nov 08, 2026 at 11:11:09PM +0800, Zhijie Hou wrote: > Thanks for the report and patch. +1. > See the attachment for a test to reproduce this issue. (Feel free to merge > if it's worth adding a test to cover this path) > I think in a rare case, this can also affect the stats collection for > spilled txns. For example, if an empty subtransaction is marked as > serialized, we skip incrementing spillTxns because no changes were written; > if real changes are later serialized in this subtransaction, we still do not > increment spillTxns because the subtransaction is already marked as > serialized and we avoid counting it "twice". Agreed. That's why I think the test is worth adding (as the bug could also produce real user visible impact as it can cause spill_txns to be undercounted on non assert enabled builds). The test only detects the regression on assert enabled builds, but I think that is enough. A test for the spill_txns case would be more complicated and is probably not worth it. > > + if (spilled) > + txn->txn_flags |= RBTXN_IS_SERIALIZED; > > ReorderBufferSerializeTXN() already has an "if (spilled)" block right above > this, so we shall directly move the assignment into that block. +1. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
Hi, On Mon, Nov 9, 2026 at 06:53 PM Bertrand Drouvot wrote: > > On Sun, Nov 08, 2026 at 11:11:09PM +0800, Zhijie Hou wrote: > > Thanks for the report and patch. > > +1. > > > See the attachment for a test to reproduce this issue. (Feel free to merge > > if it's worth adding a test to cover this path) > > > I think in a rare case, this can also affect the stats collection > > for spilled txns. For example, if an empty subtransaction is marked > > as > > serialized, we skip incrementing spillTxns because no changes were written; > > if real changes are later serialized in this subtransaction, we still do not > > increment spillTxns because the subtransaction is already marked as > > serialized and we avoid counting it "twice". > > Agreed. > > That's why I think the test is worth adding (as the bug could also > produce real user visible impact as it can cause spill_txns to be > undercounted on non assert enabled builds). The test only detects the > regression on assert enabled builds, but I think that is enough. A > test for the spill_txns case would be more complicated and is probably > not worth it. > > > > > + if (spilled) > > + txn->txn_flags |= RBTXN_IS_SERIALIZED; > > > > ReorderBufferSerializeTXN() already has an "if (spilled)" block > > right above this, so we shall directly move the assignment into that > > block. > > +1. Thanks for the review. It looks like Atisss's update patches didn't reach the list, probably a missing Reply-All, so I'm helping to attach the same patches (provided by Atisss) here. -- Best Regards, Zhijie Hou
Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
От:
Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Дата:
Hi, On Mon, Nov 09, 2026 at 07:46:59PM +0800, Zhijie Hou wrote: > Thanks for the review. It looks like Atisss's update patches didn't reach > the list, probably a missing Reply-All, so I'm helping to attach the same > patches (provided by Atisss) here. LGTM, just one nit about: + /* + * Mark the transaction as serialized only if something was written, + * and only after the check above. final_lsn is advanced for each + * change written, so a transaction that wrote nothing, such as a + * subtransaction without decoded changes, would be marked with an + * invalid final_lsn, and ReorderBufferRestoreCleanup() would later + * fail on it. + */ It looks lengthy and “the check above” is ambiguous. I'm not sure we need a new comment here, maybe just replace the existing? /* update the statistics iff we have spilled anything */ with /* update the statistics and mark the transaction serialized iff we spilled */ Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
Hi,
Really sorry for the late reply, I only saw your mail now.
Thanks for the review. v3 is attached. I dropped the long comment and
changed the existing one as you suggested:
/* update the statistics and mark the transaction serialized iff we spilled */
The code is otherwise unchanged from v2. The flag is still set at the end
of the "if (spilled)" block, after the spillTxns line that checks it.
With v3, all test_decoding regression (20) and isolation (16) tests pass on
master. The 17 and 14 backports are updated the same way, in case they help.
I did not re-run their tests after the comment change, since only the
comment differs from v2.
Regards,
Atish Prakash
Really sorry for the late reply, I only saw your mail now.
Thanks for the review. v3 is attached. I dropped the long comment and
changed the existing one as you suggested:
/* update the statistics and mark the transaction serialized iff we spilled */
The code is otherwise unchanged from v2. The flag is still set at the end
of the "if (spilled)" block, after the spillTxns line that checks it.
With v3, all test_decoding regression (20) and isolation (16) tests pass on
master. The 17 and 14 backports are updated the same way, in case they help.
I did not re-run their tests after the comment change, since only the
comment differs from v2.
Regards,
Atish Prakash
On Mon, 9 Nov 2026 at 20:17, Bertrand Drouvot <bertranddrouvot.pg@gmail.com> wrote:
Hi,
On Mon, Nov 09, 2026 at 07:46:59PM +0800, Zhijie Hou wrote:
> Thanks for the review. It looks like Atisss's update patches didn't reach
> the list, probably a missing Reply-All, so I'm helping to attach the same
> patches (provided by Atisss) here.
LGTM, just one nit about:
+ /*
+ * Mark the transaction as serialized only if something was written,
+ * and only after the check above. final_lsn is advanced for each
+ * change written, so a transaction that wrote nothing, such as a
+ * subtransaction without decoded changes, would be marked with an
+ * invalid final_lsn, and ReorderBufferRestoreCleanup() would later
+ * fail on it.
+ */
It looks lengthy and “the check above” is ambiguous. I'm not sure we need a new
comment here, maybe just replace the existing?
/* update the statistics iff we have spilled anything */
with
/* update the statistics and mark the transaction serialized iff we spilled */
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com