Re: Interesting fail when migrating Pg from Ubuntu Bionic to Focal

On Mon, Mar 07, 2022 at 09:54:22AM -0800, Andres Freund wrote:
> > Initially select didn't break anything, but when I tuned down
> > jit_above_cost so that it will kick in - got fails immediately.
> Could you set jit_debugging_support=on and show a backtrace with that?

Here you go:
Program received signal SIGSEGV, Segmentation fault.
0x0000fffe7587b6c8 in deform_0_8 ()
(gdb) bt
#0  0x0000fffe7587b6c8 in deform_0_8 ()
#1  0x0000fffe7587b5f4 in evalexpr_0_7 ()
#2  0x0000aaaab67e6f7c in ExecEvalExprSwitchContext (isNull=0xfffff7956217, econtext=0xaaaaf6014608,
state=0xaaaaf6014ab0)at ./build/../src/include/executor/executor.h:316
 
#3  ExecProject (projInfo=0xaaaaf6014aa8) at ./build/../src/include/executor/executor.h:350
#4  ExecScan (node=<optimized out>, accessMtd=0xaaaab68051b8 <SeqNext>, recheckMtd=0xaaaab6805248 <SeqRecheck>) at
./build/../src/backend/executor/execScan.c:239
#5  0x0000aaaab67ded5c in ExecProcNode (node=0xaaaaf60143b8) at ./build/../src/include/executor/executor.h:242
#6  ExecutePlan (execute_once=<optimized out>, dest=0xaaaaf5fc15e8, direction=<optimized out>, numberTuples=0,
sendTuples=<optimizedout>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0xaaaaf60143b8,
estate=0xaaaaf5e01360)
    at ./build/../src/backend/executor/execMain.c:1632
#7  standard_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=<optimized out>, count=0, execute_once=<optimized out>)
at./build/../src/backend/executor/execMain.c:350
 
#8  0x0000ffffa87751fc in pgss_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0,
execute_once=true)at ./build/../contrib/pg_stat_statements/pg_stat_statements.c:893
 
#9  0x0000ffffa875f688 in explain_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0,
execute_once=true)at ./build/../contrib/auto_explain/auto_explain.c:320
 
#10 0x0000aaaab69354c8 in PortalRunSelect (portal=portal@entry=0xaaaaf5e83ef0, forward=forward@entry=true, count=0,
count@entry=9223372036854775807,dest=dest@entry=0xaaaaf5fc15e8) at ./build/../src/backend/tcop/pquery.c:938
 
#11 0x0000aaaab6936a64 in PortalRun (portal=portal@entry=0xaaaaf5e83ef0, count=count@entry=9223372036854775807,
isTopLevel=isTopLevel@entry=true,run_once=run_once@entry=true, dest=dest@entry=0xaaaaf5fc15e8,
altdest=altdest@entry=0xaaaaf5fc15e8,
 
    completionTag=completionTag@entry=0xfffff7956848 "") at ./build/../src/backend/tcop/pquery.c:779
#12 0x0000aaaab6932a18 in exec_simple_query (query_string=query_string@entry=0xaaaaf5d9af90 "select p.proname, (SELECT
rolnamefrom pg_catalog.pg_roles where oid = p.proowner) from pg_proc p;") at
./build/../src/backend/tcop/postgres.c:1215
#13 0x0000aaaab69336ac in PostgresMain (argc=<optimized out>, argv=argv@entry=0xaaaaf5df6f18, dbname=<optimized out>,
username=<optimizedout>) at ./build/../src/backend/tcop/postgres.c:4271
 
#14 0x0000aaaab68ba768 in BackendRun (port=0xaaaaf5df0910, port=0xaaaaf5df0910) at
./build/../src/backend/postmaster/postmaster.c:4510
#15 BackendStartup (port=0xaaaaf5df0910) at ./build/../src/backend/postmaster/postmaster.c:4193
#16 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1725
#17 0x0000aaaab68bb74c in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at
./build/../src/backend/postmaster/postmaster.c:1398
#18 0x0000aaaab664c8a8 in main (argc=5, argv=0xaaaaf5d94ea0) at ./build/../src/backend/main/main.c:228

> > > FWIW, I went to the trouble of installing Ubuntu Focal on my
> > > raspberry pi to see if I could duplicate this, and I couldn't.
> > > However, what you get from a fresh install now is
> > > $ dpkg -l | grep libllvm                  
> > > ii  libllvm10:arm64                1:10.0.0-4ubuntu1                     arm64        Modular compiler and
toolchaintechnologies, runtime library
 
> > > not 9.0.1.  I also found that Fedora 31/aarch64 is still downloadable, and
> > > that does contain LLVM 9 ... and I could not reproduce it there either.
> > 
> > Soo... plot thickens.
> > 
> > Looks that pg 12 supplied by pgdg required libllvm9:
> What are the libllvm package versions on which you can reproduce the crash?

Hmm .. not sure I fully understand. Here are all packages that I think are relevant:

root@c999-pg1302:~# dpkg -l | grep -iE 'postgresql-12|llvm|clang'
ii  libllvm9:arm64                     1:9.0.1-12                            arm64        Modular compiler and
toolchaintechnologies, runtime library
 
ii  postgresql-12                      12.9-2.pgdg20.04+1                    arm64        The World's Most Advanced
OpenSource Relational Database
 
ii  postgresql-12-dbgsym               12.9-2.pgdg20.04+1                    arm64        debug symbols for
postgresql-12
ii  postgresql-12-pg-collkey           0.5.1-1insops1                        arm64        ICU collation function
wrapperfor PostgreSQL 12
 
ii  postgresql-12-postgis-2.5          2.5.5+dfsg-1.pgdg20.04+2              arm64        Geographic objects support
forPostgreSQL 12
 
ii  postgresql-12-postgis-2.5-scripts  2.5.5+dfsg-1.pgdg20.04+2              all          Geographic objects support
forPostgreSQL 12 -- SQL scripts
 
ii  postgresql-12-repack               1.4.7-2.pgdg20.04+1                   arm64        reorganize tables in
PostgreSQLdatabases with minimal locks
 
ii  postgresql-12-system-stats         1.0.0-1                               arm64        A packaged version of
EnterpriseDB'ssystem_stats postgres extension
 

Hope it helps.

Best regards,

depesz