Re: Server hangs on multiple connections

On Friday, Sep 20, 2002, at 11:30 US/Eastern, Tom Lane wrote:

> If you are using "make check" then look for
> src/test/regress/tmp_check/data/base/*/core

Thanks.  Here it is:

$ gdb src/test/regress/tmp_check/install/usr/local/pgsql/bin/postmaster
src/test/regress/tmp_check/data/base/16556/core
GNU gdb Yellow Dog Linux (5.1.1-1b)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "ppc-yellowdog-linux"...
Core was generated by `postgres: davidc regression [local] startup
                              '.
Program terminated with signal 6, Aborted.
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libhistory.so.4...done.
Loaded symbols for /usr/lib/libhistory.so.4
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
#0  0x0fd1be44 in kill () at soinit.c:76
76      soinit.c: No such file or directory.
         in soinit.c
(gdb) bt
#0  0x0fd1be44 in kill () at soinit.c:76
#1  0x0fd1bcc0 in raise (sig=6) at ../sysdeps/posix/raise.c:27
#2  0x0fd1d374 in abort () at ../sysdeps/generic/abort.c:88
#3  0x1016882c in vararg_format (fmt=0x0) at excabort.c:27
#4  0x10168744 in ExcUnCaught (excP=0x101f0968, detail=0, data=0x0,
message=0x101c16b4 "!(lock->shared > 0)") at exc.c:168
#5  0x101687d4 in ExcRaise (excP=0x101f0968, detail=0, data=0x0,
message=0x101c16b4 "!(lock->shared > 0)") at exc.c:185
#6  0x10167a4c in ExceptionalCondition (conditionName=0x101c16b4
"!(lock->shared > 0)", exceptionP=0x101f0968, detail=0x0,
     fileName=0x8 <Address 0x8 out of bounds>, lineNumber=37) at
assert.c:70
#7  0x1010ca84 in LWLockRelease (lockid=BufMgrLock) at lwlock.c:434
#8  0x10108a84 in LockAcquire (lockmethod=1, locktag=0x300eb548,
xid=806273048, lockmode=1, dontWait=0 '\000') at lock.c:723
#9  0x10107640 in LockRelation (relation=0x102507b8, lockmode=1) at
lmgr.c:153
#10 0x100333b0 in relation_openr (relationName=0x101d47a0 "pg_class",
lockmode=1) at heapam.c:524
#11 0x1003355c in heap_openr (relationName=0x0, lockmode=6) at
heapam.c:595
#12 0x101613a0 in scan_pg_rel_ind (buildinfo={infotype = 2, i =
{info_id = 270195424, info_name = 0x101adae0
"pg_trigger_tgrelid_index"}}) at relcache.c:356
#13 0x10161254 in ScanPgRelation (buildinfo={infotype = 2, i = {info_id
= 270195424, info_name = 0x101adae0 "pg_trigger_tgrelid_index"}}) at
relcache.c:284
#14 0x10162854 in RelationBuildDesc (buildinfo={infotype = 2, i =
{info_id = 270195424, info_name = 0x101adae0
"pg_trigger_tgrelid_index"}}, oldrelation=0x0)
     at relcache.c:968
#15 0x1016335c in RelationNameGetRelation (relationName=0x101adae0
"pg_trigger_tgrelid_index") at relcache.c:1493
#16 0x10033380 in relation_openr (relationName=0x101adae0
"pg_trigger_tgrelid_index", lockmode=0) at heapam.c:518
#17 0x1003ae20 in index_openr (relationName=0x0) at indexam.c:149
#18 0x1009a1a0 in RelationBuildTriggers (relation=0x301897d8) at
trigger.c:551
#19 0x1016290c in RelationBuildDesc (buildinfo={infotype = 2, i =
{info_id = 270357192, info_name = 0x101d52c8 "pg_shadow"}},
oldrelation=0x301897d8)
     at relcache.c:1033
#20 0x1016335c in RelationNameGetRelation (relationName=0x101d52c8
"pg_shadow") at relcache.c:1493
#21 0x10033380 in relation_openr (relationName=0x101d52c8 "pg_shadow",
lockmode=0) at heapam.c:518
#22 0x1003355c in heap_openr (relationName=0x0, lockmode=6) at
heapam.c:595
#23 0x1015ec20 in CatalogCacheInitializeCache (cache=0x102718b8) at
catcache.c:216
#24 0x10160194 in SearchCatCache (cache=0x102718b8, v1=270840281, v2=0,
v3=0, v4=0) at catcache.c:862
#25 0x10165c50 in SearchSysCache (cacheId=22, key1=270840281, key2=0,
key3=0, key4=0) at syscache.c:461
#26 0x1016dba4 in InitializeSessionUserId (username=0x1024b1d9
"davidc") at miscinit.c:450
#27 0x1016ea70 in InitPostgres (dbname=0x10230f40 "regression",
username=0x1024b1d9 "davidc") at postinit.c:337
#28 0x10112190 in PostgresMain (argc=4, argv=0x7fffecd8,
username=0x1024b1d9 "davidc") at postgres.c:1684
#29 0x100ecb54 in DoBackend (port=0x1024b0a8) at postmaster.c:2243
#30 0x100ec3cc in BackendStartup (port=0x1024b0a8) at postmaster.c:1874
#31 0x100eb224 in ServerLoop () at postmaster.c:977
#32 0x100eaccc in PostmasterMain (argc=4, argv=0x1022ab60) at
postmaster.c:771
#33 0x100bdf68 in main (argc=4, argv=0x7ffff814) at main.c:206
#34 0x0fd07f70 in __libc_start_main (argc=4, ubp_av=0x7ffff814,
ubp_ev=0x0, auxvec=0x7ffff8a8, rtld_fini=Cannot access memory at
address 0x0
) at ../sysdeps/powerpc/elf/libc-start.c:119


> Another interesting line of attack would be to try compiling
> src/backend/storage/lmgr/lwlock.c at different optimization levels,
> to see if the problem goes away with less optimization.  We saw a
> problem on AIX (if memory serves) before 7.2 release that turned out
> to be due to overaggressive optimization by the compiler.  We thought
> we'd added enough "volatile" keywords to lwlock.c to discourage any
> code rearrangement, but maybe we still need more.

Okay, I will try to figure out how to do what you just said :-) and
meanwhile hope the stack trace above is helpful.

Thanks!
David