Re: Server hangs on multiple connections

On Thursday, Sep 19, 2002, at 18:33 US/Eastern, Tom Lane wrote:

> David Christian <davidc@comtechmobile.com> writes:
>> Happy to.  Interestingly, when I build with --enable-debug and
>> --enable-cassert, the server doesn't lock up during 'make check', it
>> just (very quickly) fails all of the tests and exits.  I tried several
>> times.
>
> Oh, that's interesting; that says that an Assert() check is failing.
> We should investigate that first.
>
> There should be a core file left in the database subdirectory after
> the assert failure --- would you gdb it and get a stack trace from it?
> Also, you will probably find some useful messages in the postmaster
> log (which should be left in the log/ subdirectory of the regress
> tests)

Unfortunately, I see no core file under the source tree after the
assert failure.

The postmaster.log does show entries for failed assertions.  It is 246
lines long, and I am pasting it to the bottom of this message.

>> (gdb) bt
>> #0  0x0fdc297c in __syscall_ipc () at soinit.c:76
>> #1  0x0fdc38c0 in semop (semid=4, sops=0x7fffea18, nsops=1) at
>> ../sysdeps/unix/sysv/linux/semop.c:36
>> #2  0x100e4424 in IpcSemaphoreLock ()
>> #3  0x100eb018 in LWLockAcquire ()
>> #4  0x100e7f3c in LockAcquire ()
>> #5  0x100e7434 in LockRelation ()
>
> Sure enough, it would seem that everyone's stuck waiting for a lock.
> But let's chase the Assert first; that might identify the problem.

Okay, hope this helps.  I really appreciate the time you are taking to
look at this.

David


[davidc@chef ~/src/PostgreSQL/postgresql-7.2.2]$ find . -name '*core*'
./contrib/retep/uk/org/retep/xml/core
./src/interfaces/jdbc/org/postgresql/core


[davidc@chef ~/src/PostgreSQL/postgresql-7.2.2/src/test/regress/log]$
cat postmaster.log
DEBUG:  database system was shut down at 2002-09-20 02:46:51 GMT
DEBUG:  checkpoint record is at 0/113640
DEBUG:  redo record is at 0/113640; undo record is at 0/0; shutdown TRUE
DEBUG:  next transaction id: 89; next oid: 16556
DEBUG:  database system is ready
ERROR:  DROP GROUP: group "regressgroup1" does not exist
TRAP: Failed Assertion("!(lock->shared > 0):", File: "lwlock.c", Line:
434)
!(lock->shared > 0) (0) [Success]
DEBUG:  server process (pid 22628) was terminated by signal 6
DEBUG:  terminating any other active server processes
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
FATAL 1:  The database system is in recovery mode
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
DEBUG:  all server processes terminated; reinitializing shared memory
and semaphores
DEBUG:  database system was interrupted at 2002-09-20 02:46:51 GMT
DEBUG:  checkpoint record is at 0/113640
DEBUG:  redo record is at 0/113640; undo record is at 0/0; shutdown TRUE
DEBUG:  next transaction id: 89; next oid: 16556
DEBUG:  database system was not properly shut down; automatic recovery
in progress
DEBUG:  redo starts at 0/113680
DEBUG:  ReadRecord: record with zero length at 0/138818
DEBUG:  redo done at 0/1387F0
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
DEBUG:  database system is ready
ERROR:  CREATE USER: user name "regressuser4" already exists
NOTICE:  ALTER GROUP: user "regressuser2" is already in group
"regressgroup2"
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  LOCK TABLE: permission denied
ERROR:  atest2: Permission denied.
ERROR:  permission denied
ERROR:  atest2: Permission denied.
ERROR:  atest1: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest1: Permission denied.
ERROR:  atest1: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest1: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest2: Permission denied.
ERROR:  atest3: Permission denied.
ERROR:  has_table_privilege: relation "pg_shad" does not exist
ERROR:  user "nosuchuser" does not exist
ERROR:  has_table_privilege: invalid privilege type sel
ERROR:  pg_aclcheck: invalid user id 4293967297
ERROR:  has_table_privilege: invalid relation oid 1
ERROR:  Relation "onek" does not exist
ERROR:  Relation "onek" does not exist
ERROR:  Relation "tmp" does not exist
ERROR:  Relation "tmp" does not exist
ERROR:  table "tmp" does not exist
ERROR:  Relation "onek" does not exist
ERROR:  Relation "onek" does not exist
ERROR:  Relation "onek" does not exist
ERROR:  Relation "onek" does not exist
ERROR:  Relation "onek2" does not exist
ERROR:  Relation "onek2" does not exist
ERROR:  Relation "onek2" does not exist
ERROR:  Relation "stud_emp" does not exist
ERROR:  Relation "stud_emp" does not exist
ERROR:  Relation "stud_emp" does not exist
ERROR:  Relation "stud_emp" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "b_star" does not exist
ERROR:  Relation "c_star" does not exist
ERROR:  Relation "d_star" does not exist
ERROR:  Relation "e_star" does not exist
ERROR:  Relation "f_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "f_star" does not exist
ERROR:  Relation "e_star" does not exist
ERROR:  Relation "d_star" does not exist
ERROR:  Relation "c_star" does not exist
ERROR:  Relation "b_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "f_star" does not exist
ERROR:  Relation "f_star" does not exist
ERROR:  Relation "e_star" does not exist
ERROR:  Relation "e_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "a_star" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "hobbies_r" does not exist
ERROR:  Relation "hobbies_r" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Relation "person" does not exist
ERROR:  Function 'user_relns()' does not exist
         Unable to identify a function that satisfies the given argument
types
         You may need to add explicit typecasts
ERROR:  Function 'hobbies_by_name(unknown)' does not exist
         Unable to identify a function that satisfies the given argument
types
         You may need to add explicit typecasts
ERROR:  Function 'oldstyle_length(int4, text)' does not exist
         Unable to identify a function that satisfies the given argument
types
         You may need to add explicit typecasts
ERROR:  Relation "street" does not exist
ERROR:  Relation "iexit" does not exist
ERROR:  Relation "toyemp" does not exist
TRAP: Failed Assertion("!(lock->shared > 0):", File: "lwlock.c", Line:
434)
!(lock->shared > 0) (0) [Interrupted system call]
DEBUG:  server process (pid 23536) was terminated by signal 6
DEBUG:  terminating any other active server processes
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
NOTICE:  Message from PostgreSQL backend:
         The Postmaster has informed me that some other backend
         died abnormally and possibly corrupted shared memory.
         I have rolled back the current transaction and am
         going to terminate your database system connection and exit.
         Please reconnect to the database system and repeat your query.
DEBUG:  all server processes terminated; reinitializing shared memory
and semaphores
DEBUG:  database system was interrupted at 2002-09-20 02:46:54 GMT
DEBUG:  checkpoint record is at 0/138818
DEBUG:  redo record is at 0/138818; undo record is at 0/0; shutdown TRUE
DEBUG:  next transaction id: 140; next oid: 24748
DEBUG:  database system was not properly shut down; automatic recovery
in progress
DEBUG:  redo starts at 0/138858
DEBUG:  ReadRecord: record with zero length at 0/16D820
DEBUG:  redo done at 0/16D7F8
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
FATAL 1:  The database system is starting up
DEBUG:  smart shutdown request
DEBUG:  database system is ready
DEBUG:  shutting down
DEBUG:  database system is shut down