BUG #19829: BTREE pivot tuple length overflows t_info when BLCKSZ=32768

Поиск
Список
Период
Сортировка
Искать

BUG #19829: BTREE pivot tuple length overflows t_info when BLCKSZ=32768

От:
PG Bug reporting form <noreply@postgresql.org>
Дата:
The following bug has been logged on the website:

Bug reference:      19829
Logged by:          wangfei
Email address:      wf13934952799@gmail.com
PostgreSQL version: 18.6
Operating system:   ubuntu-24.04
Description:        

PostgreSQL version: 19devel (also reproducible on 18.x)
Operating system:   Linux x86_64
Configure options:  --with-blocksize=32

With BLCKSZ=32768, BTMaxItemSize() evaluates to roughly 10896 bytes, but the
tuple length field reserved by INDEX_SIZE_MASK in IndexTupleData.t_info can
only represent 8191 bytes.  index_form_tuple() enforces the 8191 limit, so a
leaf tuple is at most 8184 bytes after MAXALIGN.  _bt_truncate() may then
need
to append a tiebreaker heap TID:

    newsize = MAXALIGN(IndexTupleSize(pivot)) +
MAXALIGN(sizeof(ItemPointerData));
    ...
    tidpivot->t_info &= ~INDEX_SIZE_MASK;
    tidpivot->t_info |= newsize;

For a maximally sized leaf tuple this is 8184 + 8 = 8192, i.e. exactly
INDEX_SIZE_MASK + 1, so the length field wraps to zero.  The result is a
zero-length pivot tuple used as a leaf high key; PageAddItem() and
XLogRegisterBufData() both accept it without complaint, so the corruption is
written out and WAL-logged silently.

BTMaxItemSize() is supposed to keep leaf tuples small enough that this
cannot
happen, but the one-third-of-a-page rule scales with BLCKSZ while
INDEX_SIZE_MASK does not, so at BLCKSZ=32768 the one-third rule is no longer
the binding constraint.  Block sizes of 16384 and below are unaffected.

Steps to reproduce:
< SQL 脚本>

Expected: either the insert is rejected, or the pivot tuple is stored
correctly.
Actual:   high key with length 0 (amcheck reports: <具体报错>).

A patch against master is attached.  It takes Min() of the one-third rule
and
what t_info can represent in BTMaxItemSize(), and adds a defensive
elog(ERROR)
in _bt_truncate() so that a future miscalculation cannot corrupt an index
silently.  Since index_form_tuple() already caps index tuples at 8184 bytes,
lowering the BLCKSZ=32768 limit cannot invalidate any index that could
actually exist on disk, so there is no on-disk compatibility concern.




Re: BUG #19829: BTREE pivot tuple length overflows t_info when BLCKSZ=32768

От:
王飞 <wf13934952799@gmail.com>
Дата:
CREATE EXTENSION amcheck;

-- One incompressible 8172-byte value.  The resulting index tuple is
--   MAXALIGN(sizeof(IndexTupleData) + VARHDRSZ + 8172) = MAXALIGN(8+4+8172) = 8184
-- bytes, which is the largest index tuple index_form_tuple() will ever produce.
CREATE TABLE pad AS
SELECT substr((SELECT string_agg(md5(random()::text), '')
               FROM generate_series(1, 300)), 1, 8172) AS v;

CREATE TABLE t (v text);
-- Must come before CREATE INDEX: the index inherits attstorage, and this stops
-- TOAST_INDEX_HACK in index_form_tuple() from compressing the value.
ALTER TABLE t ALTER COLUMN v SET STORAGE EXTERNAL;
CREATE INDEX t_v_idx ON t (v);

-- Four identical keys.  Three fill the leaf page; the fourth splits it.  Since
-- all keys are equal, _bt_truncate() cannot truncate away any key attribute and
-- must append a tiebreaker heap TID: newsize = 8184 + 8 = 8192.
INSERT INTO t SELECT v FROM pad, generate_series(1, 4);

SELECT bt_index_check('t_v_idx', true);
ERROR:  invalid line pointer storage in index "t_v_idx"
DETAIL:  Index tid=(3,2) lp_off=32744, lp_len=0 lp_flags=1.

PG Bug reporting form <noreply@postgresql.org> 于2026年11月3日周二 15:47写道:
The following bug has been logged on the website:

Bug reference:      19829
Logged by:          wangfei
Email address:      wf13934952799@gmail.com
PostgreSQL version: 18.6
Operating system:   ubuntu-24.04
Description:       

PostgreSQL version: 19devel (also reproducible on 18.x)
Operating system:   Linux x86_64
Configure options:  --with-blocksize=32

With BLCKSZ=32768, BTMaxItemSize() evaluates to roughly 10896 bytes, but the
tuple length field reserved by INDEX_SIZE_MASK in IndexTupleData.t_info can
only represent 8191 bytes.  index_form_tuple() enforces the 8191 limit, so a
leaf tuple is at most 8184 bytes after MAXALIGN.  _bt_truncate() may then
need
to append a tiebreaker heap TID:

    newsize = MAXALIGN(IndexTupleSize(pivot)) +
MAXALIGN(sizeof(ItemPointerData));
    ...
    tidpivot->t_info &= ~INDEX_SIZE_MASK;
    tidpivot->t_info |= newsize;

For a maximally sized leaf tuple this is 8184 + 8 = 8192, i.e. exactly
INDEX_SIZE_MASK + 1, so the length field wraps to zero.  The result is a
zero-length pivot tuple used as a leaf high key; PageAddItem() and
XLogRegisterBufData() both accept it without complaint, so the corruption is
written out and WAL-logged silently.

BTMaxItemSize() is supposed to keep leaf tuples small enough that this
cannot
happen, but the one-third-of-a-page rule scales with BLCKSZ while
INDEX_SIZE_MASK does not, so at BLCKSZ=32768 the one-third rule is no longer
the binding constraint.  Block sizes of 16384 and below are unaffected.

Steps to reproduce:
< SQL 脚本>

Expected: either the insert is rejected, or the pivot tuple is stored
correctly.
Actual:   high key with length 0 (amcheck reports: <具体报错>).

A patch against master is attached.  It takes Min() of the one-third rule
and
what t_info can represent in BTMaxItemSize(), and adds a defensive
elog(ERROR)
in _bt_truncate() so that a future miscalculation cannot corrupt an index
silently.  Since index_form_tuple() already caps index tuples at 8184 bytes,
lowering the BLCKSZ=32768 limit cannot invalidate any index that could
actually exist on disk, so there is no on-disk compatibility concern.


Re: BUG #19829: BTREE pivot tuple length overflows t_info when BLCKSZ=32768

От:
王飞 <wf13934952799@gmail.com>
Дата:
#!/bin/bash
# =============================================================================
#  B-tree pivot tuple length overflows t_info when BLCKSZ = 32768
#
#  BTMaxItemSize() scales with BLCKSZ (10896 at 32kB pages), but the tuple
#  length field reserved by INDEX_SIZE_MASK in IndexTupleData.t_info is a fixed
#  13 bits (8191).  index_form_tuple() caps index tuples at 8184 bytes after
#  MAXALIGN, so when _bt_truncate() has to append a tiebreaker heap TID it
#  computes newsize = 8184 + 8 = 8192 = INDEX_SIZE_MASK + 1 and
#  `pivot->t_info |= newsize` wraps the length field to zero.
#
#  Needs a build with --with-blocksize=32.  At BLCKSZ 8192/16384 the
#  one-third-of-a-page rule (2704/5432) binds first, so those are unaffected.
# =============================================================================
set -u

PGBIN=${PGBIN:-/home/coder/postgres_github/release/REL_18_STABLE_32/bin}
PGDATA=${PGDATA:-/tmp/pgdata32_repro}
SOCKDIR=${SOCKDIR:-/tmp/s32repro}   # keep short: socket path limit is 107 bytes
PGPORT=${PGPORT:-55432}
PSQL="$PGBIN/psql -h $SOCKDIR -p $PGPORT -U postgres -d postgres"

echo "### 0. confirm the binary uses 32kB pages"
$PGBIN/pg_config --configure | tr ' ' '\n' | grep blocksize || {
    echo "!! not a 32kB build, cannot reproduce"; exit 1; }

echo "### 1. init + start"
rm -rf "$PGDATA"; mkdir -p "$SOCKDIR"
$PGBIN/initdb -D "$PGDATA" --locale=C --encoding=UTF8 -U postgres >/dev/null 2>&1
cat >> "$PGDATA/postgresql.conf" <<EOF
port = $PGPORT
listen_addresses = ''
unix_socket_directories = '$SOCKDIR'
fsync = off
restart_after_crash = off
EOF
$PGBIN/pg_controldata -D "$PGDATA" | grep -i "Database block size"
$PGBIN/pg_ctl -D "$PGDATA" -l "$PGDATA/server.log" start -w >/dev/null 2>&1

echo "### 2. build the corrupt index (this is the whole repro)"
$PSQL -v ON_ERROR_STOP=1 <<'EOF'
CREATE EXTENSION amcheck;
CREATE EXTENSION pageinspect;

-- One incompressible 8172-byte value.  The resulting index tuple is
--   MAXALIGN(sizeof(IndexTupleData) + VARHDRSZ + 8172) = MAXALIGN(8+4+8172)
--   = 8184 bytes
-- which is the largest index tuple index_form_tuple() will ever produce.
CREATE TABLE pad AS
SELECT substr((SELECT string_agg(md5(random()::text), '')
               FROM generate_series(1, 300)), 1, 8172) AS v;

CREATE TABLE t (v text);
-- Must come before CREATE INDEX: the index inherits attstorage, and this stops
-- TOAST_INDEX_HACK in index_form_tuple() from compressing the value.
ALTER TABLE t ALTER COLUMN v SET STORAGE EXTERNAL;
CREATE INDEX t_v_idx ON t (v);
CHECKPOINT;

-- Four identical keys.  Three fill the leaf page; the fourth splits it.  Since
-- all the keys are equal, _bt_truncate() cannot truncate away any key
-- attribute and must append a tiebreaker heap TID: newsize = 8184 + 8 = 8192.
INSERT INTO t SELECT v FROM pad, generate_series(1, 4);
EOF

echo
echo "### 3. amcheck"
$PSQL -c "SELECT bt_index_check('t_v_idx', true);"        2>&1 | sed 's/^/  /'
$PSQL -c "SELECT bt_index_parent_check('t_v_idx', true);" 2>&1 | sed 's/^/  /'

echo "### 4. page layout: zero-length pivot as leaf high key AND as root downlink"
$PSQL -c "SELECT b AS blkno, s.type, s.live_items, s.avg_item_size
          FROM generate_series(1,3) b, bt_page_stats('t_v_idx', b) s;"
$PSQL -c "
WITH g AS (SELECT blk, get_raw_page('t_v_idx', blk) AS pg FROM generate_series(1,3) blk),
     w AS (SELECT blk, off,
                  get_byte(pg,24+(off-1)*4) + get_byte(pg,25+(off-1)*4)*256
                + get_byte(pg,26+(off-1)*4)*65536 + get_byte(pg,27+(off-1)*4)*16777216 AS word
           FROM g, generate_series(1,3) off)
SELECT blk, off, word & 32767 AS lp_off, (word/32768) & 3 AS lp_flags, word/131072 AS lp_len
FROM w ORDER BY blk, off;"

echo "### 5. pageinspect underflows on the zero-length item"
$PSQL -Atc "SELECT itemlen FROM bt_page_items('t_v_idx', 1);" 2>&1 | head -2 | sed 's/^/  /'

echo "### 6. silently wrong query results"
echo -n "  indexscan: "; $PSQL -Atc \
  "SET enable_seqscan=off; SELECT count(*) FROM t WHERE v = (SELECT v FROM pad);"
echo -n "  seqscan:   "; $PSQL -Atc \
  "SET enable_indexscan=off; SET enable_bitmapscan=off;
   SELECT count(*) FROM t WHERE v = (SELECT v FROM pad);"

echo "### 7. the split record is WAL-logged without a full-page image"
$PSQL -Atc "SELECT pg_switch_wal();" >/dev/null
for f in $(ls "$PGDATA/pg_wal" | grep -E '^[0-9A-F]{24}$'); do
    $PGBIN/pg_waldump -p "$PGDATA/pg_wal" "$f" 2>/dev/null | grep "SPLIT_R"
done | tail -1 | sed 's/^/  /'

echo "### 8. crash + restart => recovery segfaults, cluster will not come up"
$PGBIN/pg_ctl -D "$PGDATA" stop -m immediate >/dev/null 2>&1
: > "$PGDATA/server.log"
$PGBIN/pg_ctl -D "$PGDATA" -l "$PGDATA/server.log" start -w -t 20 >/dev/null 2>&1
sed 's/^/  /' "$PGDATA/server.log"

echo "### 9. gdb backtrace of the recovery crash"
if command -v gdb >/dev/null; then
    cd "$SOCKDIR" && timeout 180 gdb --batch -q \
        -ex 'run' -ex 'bt 4' \
        -ex 'p $_siginfo._sifields._sigfault.si_addr' \
        --args "$PGBIN/postgres" --single -D "$PGDATA" postgres </dev/null 2>&1 \
        | grep -E "SIGSEGV|^#|\\\$1" | sed 's/^/  /'
fi


王飞 <wf13934952799@gmail.com> 于2026年11月3日周二 16:10写道:
CREATE EXTENSION amcheck;

-- One incompressible 8172-byte value.  The resulting index tuple is
--   MAXALIGN(sizeof(IndexTupleData) + VARHDRSZ + 8172) = MAXALIGN(8+4+8172) = 8184
-- bytes, which is the largest index tuple index_form_tuple() will ever produce.
CREATE TABLE pad AS
SELECT substr((SELECT string_agg(md5(random()::text), '')
               FROM generate_series(1, 300)), 1, 8172) AS v;

CREATE TABLE t (v text);
-- Must come before CREATE INDEX: the index inherits attstorage, and this stops
-- TOAST_INDEX_HACK in index_form_tuple() from compressing the value.
ALTER TABLE t ALTER COLUMN v SET STORAGE EXTERNAL;
CREATE INDEX t_v_idx ON t (v);

-- Four identical keys.  Three fill the leaf page; the fourth splits it.  Since
-- all keys are equal, _bt_truncate() cannot truncate away any key attribute and
-- must append a tiebreaker heap TID: newsize = 8184 + 8 = 8192.
INSERT INTO t SELECT v FROM pad, generate_series(1, 4);

SELECT bt_index_check('t_v_idx', true);
ERROR:  invalid line pointer storage in index "t_v_idx"
DETAIL:  Index tid=(3,2) lp_off=32744, lp_len=0 lp_flags=1.

PG Bug reporting form <noreply@postgresql.org> 于2026年11月3日周二 15:47写道:
The following bug has been logged on the website:

Bug reference:      19829
Logged by:          wangfei
Email address:      wf13934952799@gmail.com
PostgreSQL version: 18.6
Operating system:   ubuntu-24.04
Description:       

PostgreSQL version: 19devel (also reproducible on 18.x)
Operating system:   Linux x86_64
Configure options:  --with-blocksize=32

With BLCKSZ=32768, BTMaxItemSize() evaluates to roughly 10896 bytes, but the
tuple length field reserved by INDEX_SIZE_MASK in IndexTupleData.t_info can
only represent 8191 bytes.  index_form_tuple() enforces the 8191 limit, so a
leaf tuple is at most 8184 bytes after MAXALIGN.  _bt_truncate() may then
need
to append a tiebreaker heap TID:

    newsize = MAXALIGN(IndexTupleSize(pivot)) +
MAXALIGN(sizeof(ItemPointerData));
    ...
    tidpivot->t_info &= ~INDEX_SIZE_MASK;
    tidpivot->t_info |= newsize;

For a maximally sized leaf tuple this is 8184 + 8 = 8192, i.e. exactly
INDEX_SIZE_MASK + 1, so the length field wraps to zero.  The result is a
zero-length pivot tuple used as a leaf high key; PageAddItem() and
XLogRegisterBufData() both accept it without complaint, so the corruption is
written out and WAL-logged silently.

BTMaxItemSize() is supposed to keep leaf tuples small enough that this
cannot
happen, but the one-third-of-a-page rule scales with BLCKSZ while
INDEX_SIZE_MASK does not, so at BLCKSZ=32768 the one-third rule is no longer
the binding constraint.  Block sizes of 16384 and below are unaffected.

Steps to reproduce:
< SQL 脚本>

Expected: either the insert is rejected, or the pivot tuple is stored
correctly.
Actual:   high key with length 0 (amcheck reports: <具体报错>).

A patch against master is attached.  It takes Min() of the one-third rule
and
what t_info can represent in BTMaxItemSize(), and adds a defensive
elog(ERROR)
in _bt_truncate() so that a future miscalculation cannot corrupt an index
silently.  Since index_form_tuple() already caps index tuples at 8184 bytes,
lowering the BLCKSZ=32768 limit cannot invalidate any index that could
actually exist on disk, so there is no on-disk compatibility concern.


FAQ