Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

On Tue, Jul 28, 2020 at 11:10 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> An internal server aborted last night while running a maintenance script.  I
> reproduced this easily running the crashing command in a loop, and verified
> this is a live issue on REL_13_STABLE (dc6f2fb43).
> REINDEX INDEX pg_class_tblspc_relfilenode_index
>
> It looks like this crashed once before, and I didn't notice until now:
> Fri Jun 26 22:30:29 CDT 2020: pg_shdescription: pg_toast.pg_toast_2396_index(reindex toast)...
> psql: error: could not connect to server: server closed the connection unexpectedly
>
> #3  0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
> 4251            Assert(false);
> (gdb) l
> 4246                    if (pos1 != pos2)
> 4247                            return (pos1 < pos2) ? -1 : 1;
> 4248            }
> 4249
> 4250            /* ItemPointer values should never be equal */
> 4251            Assert(false);
> 4252
> 4253            return 0;
> 4254    }
> 4255
>
> #3  0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
>         sortKey = 0x2014d60
>         tuple1 = 0x20189d8
>         tuple2 = 0x2018898
>         keysz = 2
>         tupDes = 0x7f48996b3790
>         equal_hasnull = false
>         nkey = 3
>         compare = 0
>         datum1 = 67999603
>         datum2 = 67999603
>         isnull1 = false
>         isnull2 = false
>         __func__ = "comparetup_index_btree"
>
> (gdb) p *tuple1
> $2 = {t_tid = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 43}, t_info = 16}
> (gdb) p *tuple2
> $3 = {t_tid = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 43}, t_info = 16}
>
> (gdb) p *sortKey
> $5 = {ssup_cxt = 0x40, ssup_collation = 0, ssup_reverse = false, ssup_nulls_first = false, ssup_attno = 0, ssup_extra
=0x0, comparator = 0x7f7f7f7f7f7f7f7f, abbreviate = 127, abbrev_converter = 0x7f7f7f7f7f7f7f7f,
 
>   abbrev_abort = 0x7f7f7f7f7f7f7f7f, abbrev_full_comparator = 0x7f7f7f7f7f7f7f7f}
>
> (gdb) p *tupDes
> $6 = {natts = 2, tdtypeid = 0, tdtypmod = -1, tdrefcount = 1, constr = 0x0, attrs = 0x7f48996b37a8}
>
> Core was generated by `postgres: postgres sentinel [local] REINDEX            '.
>
> (gdb) bt
> #0  0x00007f489853d1f7 in raise () from /lib64/libc.so.6
> #1  0x00007f489853e8e8 in abort () from /lib64/libc.so.6
> #2  0x0000000000aafff7 in ExceptionalCondition (conditionName=0xccd0dc "false", errorType=0xccc327 "FailedAssertion",
fileName=0xccc2fd"tuplesort.c", lineNumber=4251) at assert.c:67
 
> #3  0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
> #4  0x0000000000af1d5e in qsort_tuple (a=0x201fa10, n=18, cmp_tuple=0xafcf21 <comparetup_index_btree>,
state=0x20147b0)at qsort_tuple.c:140
 
> #5  0x0000000000af2104 in qsort_tuple (a=0x201f710, n=50, cmp_tuple=0xafcf21 <comparetup_index_btree>,
state=0x20147b0)at qsort_tuple.c:191
 
> #6  0x0000000000af2104 in qsort_tuple (a=0x201cc38, n=544, cmp_tuple=0xafcf21 <comparetup_index_btree>,
state=0x20147b0)at qsort_tuple.c:191
 
> #7  0x0000000000af8056 in tuplesort_sort_memtuples (state=0x20147b0) at tuplesort.c:3490
> #8  0x0000000000af51a9 in tuplesort_performsort (state=0x20147b0) at tuplesort.c:1985
> #9  0x0000000000529418 in _bt_leafbuild (btspool=0x1f784e0, btspool2=0x0) at nbtsort.c:553
> #10 0x0000000000528f9c in btbuild (heap=0x1fb5758, index=0x7f48996b3460, indexInfo=0x1f77a48) at nbtsort.c:333
> #11 0x00000000005adcb3 in index_build (heapRelation=0x1fb5758, indexRelation=0x7f48996b3460, indexInfo=0x1f77a48,
isreindex=true,parallel=true) at index.c:2903
 
> #12 0x00000000005aec6b in reindex_index (indexId=3455, skip_constraint_checks=false, persistence=112 'p', options=2)
atindex.c:3539
 
> #13 0x0000000000692583 in ReindexIndex (indexRelation=0x1f54840, options=0, concurrent=false) at indexcmds.c:2466
> #14 0x0000000000932e36 in standard_ProcessUtility (pstmt=0x1f54960, queryString=0x1f53d90 "REINDEX INDEX
pg_class_tblspc_relfilenode_index",context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1f54c40,
 
>     qc=0x7ffde023bf80) at utility.c:929
> #15 0x000000000093241f in ProcessUtility (pstmt=0x1f54960, queryString=0x1f53d90 "REINDEX INDEX
pg_class_tblspc_relfilenode_index",context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1f54c40,
qc=0x7ffde023bf80)
>     at utility.c:524
> #16 0x0000000000931288 in PortalRunUtility (portal=0x1fb5ac0, pstmt=0x1f54960, isTopLevel=true,
setHoldSnapshot=false,dest=0x1f54c40, qc=0x7ffde023bf80) at pquery.c:1157
 
> #17 0x00000000009314a7 in PortalRunMulti (portal=0x1fb5ac0, isTopLevel=true, setHoldSnapshot=false, dest=0x1f54c40,
altdest=0x1f54c40,qc=0x7ffde023bf80) at pquery.c:1303
 
> #18 0x00000000009309bc in PortalRun (portal=0x1fb5ac0, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x1f54c40,altdest=0x1f54c40, qc=0x7ffde023bf80) at pquery.c:779
 
> #19 0x000000000092ab5b in exec_simple_query (query_string=0x1f53d90 "REINDEX INDEX
pg_class_tblspc_relfilenode_index")at postgres.c:1239
 
> #20 0x000000000092eb82 in PostgresMain (argc=1, argv=0x1f7db80, dbname=0x1f509d8 "sentinel", username=0x1f7daa0
"pryzbyj")at postgres.c:4315
 
> #21 0x000000000087f098 in BackendRun (port=0x1f75a80) at postmaster.c:4523
> #22 0x000000000087e888 in BackendStartup (port=0x1f75a80) at postmaster.c:4215
> #23 0x000000000087ae95 in ServerLoop () at postmaster.c:1727
> #24 0x000000000087a76c in PostmasterMain (argc=5, argv=0x1f4e8e0) at postmaster.c:1400
> #25 0x00000000007823f3 in main (argc=5, argv=0x1f4e8e0) at main.c:210
>
> This appears to be an issue with d2d8a229b (Incremental Sort), so I will add
> at: https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items

Is that assumption largely based on the incremental sort patch
refactoring tuplesort.c a bit? I haven't looked at it much at all, but
I'm wondering if the issue could also be related to the btree
duplicates changes in 13 given that we're looking at
comparetup_index_btree and the datums are equal.

James