crash in ltree_compare_distance() in 14.24
crash in ltree_compare_distance() in 14.24
I don't have it compiled with debug, and when I recompile and try the old core I get corrupted stack, but hopefully you can make some use of this data as is. root@Satsuki:/var/db/postgres/data14 # gdb /usr/local/bin/postgres postgres.core GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] Reading symbols from /usr/local/bin/postgres... (No debugging symbols found in /usr/local/bin/postgres) [New LWP 101430] Core was generated by `postgres: '. Program terminated with signal SIGSEGV, Segmentation fault. Address not mapped to object. #0 0x0000001269bfe52a in ltree_compare_distance () from /usr/local/lib/postgresql/ltree.so (gdb) (gdb) bt #0 0x0000001269bfe52a in ltree_compare_distance () from /usr/local/lib/postgresql/ltree.so #1 0x0000001269bfb74b in ltree_penalty () from /usr/local/lib/postgresql/ltree.so #2 0x0000000000962cad in FunctionCall3Coll () #3 0x00000000005104eb in gistchoose () #4 0x000000000050529d in gistdoinsert () #5 0x0000000000504b7b in gistinsert () #6 0x0000000000694322 in ExecInsertIndexTuples () #7 0x00000000006c2485 in ?? () #8 0x00000000006c13ee in ?? () #9 0x0000000000695a2b in standard_ExecutorRun () #10 0x000000082bad035f in ?? () from /usr/local/lib/postgresql/pg_stat_statements.so #11 0x00000000008231bf in ?? () #12 0x0000000000822892 in ?? () #13 0x0000000000822330 in PortalRun () #14 0x000000000081fffe in PostgresMain () #15 0x0000000000792b42 in ?? () #16 0x00000000007917bc in ?? () #17 0x000000000078eaf4 in PostmasterMain () #18 0x00000000006e9ee1 in main () (gdb) info registers rax 0x3f9ff300 1067447040 rbx 0x0 0 rcx 0x8 8 rdx 0x0 0 rsi 0x127a2f2a0f 79359322639 rdi 0x2fa39243b1d8 52379580084696 rbp 0x8212247d0 0x8212247d0 rsp 0x821224790 0x821224790 r8 0x82122491c 34915633436 r9 0x0 0 r10 0x2fa392591378 52379581485944 r11 0x123a8f3ca0 78291877024 r12 0x8 8 r13 0x2fa39243b1d0 52379580084688 r14 0x127a2f2a0f 79359322639 r15 0x2fa39243b1d8 52379580084696 rip 0x1269bfe52a 0x1269bfe52a eflags 0x10246 [ PF ZF IF RF ] cs 0x43 67 ss 0x3b 59 ds 0x3b 59 es 0x3b 59 fs 0x13 19 gs 0x1b 27 fs_base 0x2fa3916573a0 52379565519776 gs_base 0x0 0 (gdb) disassemble Dump of assembler code for function ltree_compare_distance: 0x0000001269bfe510 <+0>: push %rbp 0x0000001269bfe511 <+1>: mov %rsp,%rbp 0x0000001269bfe514 <+4>: push %r15 0x0000001269bfe516 <+6>: push %r14 0x0000001269bfe518 <+8>: push %r13 0x0000001269bfe51a <+10>: push %r12 0x0000001269bfe51c <+12>: push %rbx 0x0000001269bfe51d <+13>: sub $0x18,%rsp 0x0000001269bfe521 <+17>: movzwl 0x4(%rdi),%edx 0x0000001269bfe525 <+21>: test %edx,%edx 0x0000001269bfe527 <+23>: setne %al => 0x0000001269bfe52a <+26>: movzwl 0x4(%rsi),%ecx 0x0000001269bfe52e <+30>: movzwl %cx,%ecx 0x0000001269bfe531 <+33>: mov %ecx,-0x2c(%rbp) 0x0000001269bfe534 <+36>: test %cx,%cx 0x0000001269bfe537 <+39>: setne %cl 0x0000001269bfe53a <+42>: and %al,%cl 0x0000001269bfe53c <+44>: mov %edx,-0x30(%rbp) 0x0000001269bfe53f <+47>: mov %edx,%eax 0x0000001269bfe541 <+49>: cmp $0x1,%cl 0x0000001269bfe544 <+52>: jne 0x1269bfe5b8 0x0000001269bfe546 <+54>: mov %rsi,%rbx 0x0000001269bfe549 <+57>: mov %rdi,%r14 0x0000001269bfe54c <+60>: add $0x8,%rbx 0x0000001269bfe550 <+64>: add $0x8,%r14 0x0000001269bfe554 <+68>: mov -0x2c(%rbp),%eax 0x0000001269bfe557 <+71>: mov %eax,%r12d 0x0000001269bfe55a <+74>: mov -0x30(%rbp),%eax 0x0000001269bfe55d <+77>: mov %eax,%edx 0x0000001269bfe55f <+79>: nop 0x0000001269bfe560 <+80>: mov %rdx,-0x38(%rbp) 0x0000001269bfe564 <+84>: lea 0x2(%r14),%rdi 0x0000001269bfe568 <+88>: lea 0x2(%rbx),%rsi 0x0000001269bfe56c <+92>: movzwl (%r14),%r15d 0x0000001269bfe570 <+96>: movzwl (%rbx),%r13d 0x0000001269bfe574 <+100>: cmp %r13w,%r15w 0x0000001269bfe578 <+104>: mov %r13d,%edx 0x0000001269bfe57b <+107>: cmovb %r15d,%edx 0x0000001269bfe57f <+111>: call 0x1269c01310 0x0000001269bfe584 <+116>: test %eax,%eax 0x0000001269bfe586 <+118>: jne 0x1269bfe5d9 0x0000001269bfe588 <+120>: cmp %r13w,%r15w 0x0000001269bfe58c <+124>: mov -0x38(%rbp),%rcx 0x0000001269bfe590 <+128>: jne 0x1269bfe5f1 0x0000001269bfe592 <+130>: lea -0x1(%rcx),%eax 0x0000001269bfe595 <+133>: cmp $0x2,%ecx 0x0000001269bfe598 <+136>: jl 0x1269bfe5b8 0x0000001269bfe59a <+138>: lea -0x1(%r12),%ecx 0x0000001269bfe59f <+143>: add $0x9,%r15d 0x0000001269bfe5a3 <+147>: and $0xfffffff8,%r15d 0x0000001269bfe5a7 <+151>: add %r15,%r14 0x0000001269bfe5aa <+154>: add %r15,%rbx 0x0000001269bfe5ad <+157>: mov %eax,%edx 0x0000001269bfe5af <+159>: cmp $0x1,%r12d 0x0000001269bfe5b3 <+163>: mov %ecx,%r12d 0x0000001269bfe5b6 <+166>: jg 0x1269bfe560 0x0000001269bfe5b8 <+168>: mov -0x30(%rbp),%ecx 0x0000001269bfe5bb <+171>: sub -0x2c(%rbp),%ecx 0x0000001269bfe5be <+174>: cvtsi2ss %ecx,%xmm0 0x0000001269bfe5c2 <+178>: cvtss2sd %xmm0,%xmm1
Re: crash in ltree_compare_distance() in 14.24
Hi Chris! > I don't have it compiled with debug, and when I recompile and try the > old core I get corrupted stack, but hopefully you can make some use of > this data as is. Can you reproduce this and share the steps? In case you cannot reproduce it, can you share at least the query you were running and the relevant parts of the schema? -- David Geier
Re: crash in ltree_compare_distance() in 14.24
Hi Chris!I don't have it compiled with debug, and when I recompile and try the old core I get corrupted stack, but hopefully you can make some use of this data as is.Can you reproduce this and share the steps? In case you cannot reproduce it, can you share at least the query you were running and the relevant parts of the schema?
It happened in the middle of the night, has not happened since.
The ltree column id_tree contains a ltree of comment ID's showing the tree of comment replies.
Logs show this
2026-10-26 02:26:35.914 PDT [61206] LOG: server process (PID 50023) was terminated by signal 11: Segmentation fault
2026-10-26 02:26:35.914 PDT [61206] DETAIL: Failed process was running: INSERT INTO "story_comments" ("comment", "reply_to", "deleted", "id_tree", "sid", "user_id", "ipaddr") VALUES ($1, $2, $3, $4, $5, $6, $7)
Due to parameterised queries, I do not have the data values that were inserted.
Here is the table structure.
database=# \d story_comments
Table "public.story_comments"
Column | Type | Collation | Nullable | Default
--------------+-----------------------------+-----------+----------+----------------------------------------------------
comment_id | integer | | not null | nextval('story_comments_comment_id_seq'::regclass)
user_id | integer | | not null |
comment | text | | not null | ''::text
comment_date | timestamp without time zone | | not null | now()
ipaddr | inet | | not null |
reply_to | integer | | not null | 0
deleted | smallint | | not null | 0
id_tree | ltree | | not null | ''::ltree
reply_date | timestamp with time zone | | | now()
sid | integer | | not null |
chapter | integer | | not null | 0
Indexes:
"story_comments_pkey" PRIMARY KEY, btree (comment_id)
"idx_story_comments_comment_date" btree (comment_date)
"idx_story_comments_sid" btree (sid) CLUSTER
"idx_story_comments_user_id" btree (user_id)
"story_comments_undeleted" gist (id_tree, sid) WHERE deleted = 0
Foreign-key constraints:
"story_comments_sid_fkey" FOREIGN KEY (sid) REFERENCES stories(sid)
"story_comments_sid_fkey1" FOREIGN KEY (sid) REFERENCES stories(sid)
"story_comments_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(user_id)
"story_comments_user_id_fkey1" FOREIGN KEY (user_id) REFERENCES users(user_id)
Triggers:
update_story_comments_count BEFORE INSERT OR DELETE ON story_comments FOR EACH ROW EXECUTE FUNCTION update_story_comments_count()
update_story_comments_ltree AFTER INSERT ON story_comments FOR EACH ROW EXECUTE FUNCTION update_story_comment_ltrees()
update_story_comments_reply_date AFTER INSERT ON story_comments FOR EACH ROW EXECUTE FUNCTION update_comments_reply_date('story')
update_user_comments_count BEFORE INSERT OR DELETE ON story_comments FOR EACH ROW EXECUTE FUNCTION update_user_comments_count()
Inherits: base_comments
database=# \d base_comments
Table "public.base_comments"
Column | Type | Collation | Nullable | Default
--------------+-----------------------------+-----------+----------+---------------------------------------------------
comment_id | integer | | not null | nextval('base_comments_comment_id_seq'::regclass)
user_id | integer | | not null |
comment | text | | not null | ''::text
comment_date | timestamp without time zone | | not null | now()
ipaddr | inet | | not null |
reply_to | integer | | not null | 0
deleted | smallint | | not null | 0
id_tree | ltree | | not null | ''::ltree
reply_date | timestamp with time zone | | | now()
Indexes:
"base_comments_pkey" PRIMARY KEY, btree (comment_id)
Foreign-key constraints:
"base_comments_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(user_id)
Number of child tables: 4 (Use \d+ to list them.)
database=# \sf update_story_comment_ltrees
CREATE OR REPLACE FUNCTION public.update_story_comment_ltrees()
RETURNS trigger
LANGUAGE plpgsql
AS $function$BEGIN
PERFORM generate_story_comment_ltrees(NEW.comment_id);
return NEW;
END;$function$database=# \sf generate_story_comment_ltrees
CREATE OR REPLACE FUNCTION public.generate_story_comment_ltrees(c integer)
RETURNS character varying
LANGUAGE plpgsql
AS $function$DECLARE
id_hier text = '';
myrec RECORD;
BEGIN
SELECT INTO myrec reply_to, comment_id from story_comments where comment_id = c;
IF NOT FOUND THEN
RAISE EXCEPTION 'comment % not found', c;
return '';
END IF;
LOOP
IF id_hier != '' THEN
id_hier := myrec.comment_id || '.' || id_hier;
ELSE
id_hier := myrec.comment_id;
END IF;
-- RAISE NOTICE 'id_hier % reply_to %', id_hier, myrec.reply_to;
EXIT WHEN myrec.reply_to = 0;
SELECT INTO myrec reply_to, comment_id from story_comments where comment_id = myrec.reply_to;
IF NOT FOUND THEN
RAISE NOTICE 'Comment % reply_to % not found', c, myrec.reply_to;
RAISE EXCEPTION '';
return '';
END IF;
END LOOP;
-- RAISE NOTICE 'id_hier %', id_hier;
update story_comments set id_tree=text2ltree(id_hier) where comment_id = c and id_tree != text2ltree(id_hier);
return id_hier;
return null;
END;$function$
Re: crash in ltree_compare_distance() in 14.24
>> In case you cannot reproduce it, can you share at least the query you
>> were running and the relevant parts of the schema?
>
> It happened in the middle of the night, has not happened since.
>
> The ltree column id_tree contains a ltree of comment ID's showing the
> tree of comment replies.
>
> Logs show this
>
> 2026-10-26 02:26:35.914 PDT [61206] LOG: server process (PID
> 50023) was terminated by signal 11: Segmentation fault
> 2026-10-26 02:26:35.914 PDT [61206] DETAIL: Failed process was
> running: INSERT INTO "story_comments" ("comment", "reply_to",
> "deleted", "id_tree", "sid", "user_id", "ipaddr") VALUES ($1, $2,
> $3, $4, $5, $6, $7)
>
> Due to parameterised queries, I do not have the data values that were
> inserted.
>
> Here is the table structure.
Thanks for sharing the details.
What can be concluded from the core dump information you shared in your
first mail is that the read of b->numlevel in
float
ltree_compare_distance(const ltree *a, const ltree *b)
{
ltree_level *al = LTREE_FIRST(a);
ltree_level *bl = LTREE_FIRST(b);
int an = a->numlevel;
int bn = b->numlevel;
causes the segfault, likely because "b" is a corrupted pointer. The
value of b is 0x127a2f2a13, which is unaligned and far away from "a"
which is 0x2fa39243b1d8 and aligned as expected. Beyond that, the "a"
ltree is empty / has 0 levels as can be inferred from edx = 0.
I tried to create a failing test case based on your schema details but
failed. I'm assuming that the order of INSERTs and the data matters.
What you could try is open your core dump and go down the call stack to
see where "b" gets messed up. It would also be interesting which of the
two calls to ltree_compare_distance() in ltree_penalty() causes the
segfault.
Do you use data checksums? If not, maybe the issue stems from a data
corruption on disk?
FYI: PostgreSQL 14 reaches end of life on November 12, 2026. Though, I
think it's not unlikely that this bug is also on master.
--
David Geier
Re: crash in ltree_compare_distance() in 14.24
On 2026-10-28 12:19 p.m., David Geier wrote:
>>> In case you cannot reproduce it, can you share at least the query you
>>> were running and the relevant parts of the schema?
>> It happened in the middle of the night, has not happened since.
>>
>> The ltree column id_tree contains a ltree of comment ID's showing the
>> tree of comment replies.
>>
>> Logs show this
>>
>> 2026-10-26 02:26:35.914 PDT [61206] LOG: server process (PID
>> 50023) was terminated by signal 11: Segmentation fault
>> 2026-10-26 02:26:35.914 PDT [61206] DETAIL: Failed process was
>> running: INSERT INTO "story_comments" ("comment", "reply_to",
>> "deleted", "id_tree", "sid", "user_id", "ipaddr") VALUES ($1, $2,
>> $3, $4, $5, $6, $7)
>>
>> Due to parameterised queries, I do not have the data values that were
>> inserted.
>>
>> Here is the table structure.
> Thanks for sharing the details.
>
> What can be concluded from the core dump information you shared in your
> first mail is that the read of b->numlevel in
>
> float
> ltree_compare_distance(const ltree *a, const ltree *b)
> {
> ltree_level *al = LTREE_FIRST(a);
> ltree_level *bl = LTREE_FIRST(b);
> int an = a->numlevel;
> int bn = b->numlevel;
>
> causes the segfault, likely because "b" is a corrupted pointer. The
> value of b is 0x127a2f2a13, which is unaligned and far away from "a"
> which is 0x2fa39243b1d8 and aligned as expected. Beyond that, the "a"
> ltree is empty / has 0 levels as can be inferred from edx = 0.
>
> I tried to create a failing test case based on your schema details but
> failed. I'm assuming that the order of INSERTs and the data matters.
>
> What you could try is open your core dump and go down the call stack to
> see where "b" gets messed up. It would also be interesting which of the
> two calls to ltree_compare_distance() in ltree_penalty() causes the
> segfault.
>
> Do you use data checksums? If not, maybe the issue stems from a data
> corruption on disk?
>
> FYI: PostgreSQL 14 reaches end of life on November 12, 2026. Though, I
> think it's not unlikely that this bug is also on master.
Data corruption is possible, I recently had an incident with it, however
my thinking was that if this was caused by corruption that the pg_dump
would also crash the server which is not happening. That assumption may
be incorrect.
If corruption is the cause, can you give guidance on what I should do to
repair/validate that data? Which index should be rebuilt/etc? I will be
doing a full dump/restore to upgrade to latest version soonish (need to
validate that new version won't break anything in the application) which
should hopefully remove any issues there.
I do not know gdb very well, so I will need guidance for commands to 'go
down the call stack tosee where "b" gets messed up'. I recompiled
postgres to try to get debugging symbols, and now backtrace is corrupt
so I don't know if I can still get useful data out of the core. I guess
I can go back to the old binary if the data is useful, but as you said
it's an old version.
Unfortunately not using checksums.
Re: crash in ltree_compare_distance() in 14.24
Chris St Denis writes: > Data corruption is possible, I recently had an incident with it, however > my thinking was that if this was caused by corruption that the pg_dump > would also crash the server which is not happening. That assumption may > be incorrect. pg_dump is pretty insensitive to corruption in user-table indexes. It normally reads user tables only via full-table seqscans, which won't touch the indexes at all. In this case, we can assume that ltree_compare_distance is being asked to compute the distance between some ltree value in the index and one in local memory (a value to be inserted), and it seems it's being passed an incorrect pointer to the index entry, implying corruption in the index structure. > If corruption is the cause, can you give guidance on what I should do to > repair/validate that data? Which index should be rebuilt/etc? If you don't have a way to reproduce it, then there's not a lot we can do to investigate. I'd advise just reindexing the troublesome index and moving on. If you're unsure which is the troublesome index, reindexing all GIST indexes on ltree columns should be enough to deal with it. regards, tom lane