i'm hoping someone might be able to help me understand some of what
might been going on in the environment external to postgres based on
some error messages i just got in the logs of one of my pg installations.
in a period of duress (i.e., the box itself was rebooting and postgres
was dying) on one of my servers, i saw the following a few times:
ERROR: deadlock detected
shortly afterward, there was this:
DEBUG: database system was interrupted at <timestamp>
what can interrupt the database? i've seen it get terminated by signal 9
when the box is failing, but is there any way to know what might have
interrupted it? it seems to have died altogether, as it then goes
through the checkpoint record, etc.
after it starts up, this:
DEBUG: database system is ready
DEBUG: server process (pid 882) was terminated by signal 11
does this mean postgres itself segfaulted or that it received an
external SIGSEGV from a critical system process (e.g., the kernel)?
also, is there any significance to the fact that those two statements
occurred one right after the other?
a little later, after another recovery, i see these:
DEBUG: all server processes terminated; reinitializing shared memory
and semaphores
DEBUG: database system was interrupted at <timestamp>
finally, it seems to stabilize for a bit. then, a little later, a whole
spew of garbage characters in the log immediately preceding another:
DEBUG: database system was interrupted at <timestamp>
how would garbage data end up in the log? does that indicate anything
about the manner in which postgres was interrupted?
after yet another recovery, i see the following:
FATAL 1: The database system is starting up
FATAL 2: open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2: open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2: open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2: open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2: open of $PGDATA/pg_clog/0419 failed: No such file or directory
DEBUG: server process (pid 945) exited with exit code 2
here, what is exit code 2? does that just mean that postgres found a
significant problem with clog files?
then, there was one more of the shared memory/interruption pairs, a
final recovery, and smooth sailing again.
i guess i'd like to be able to determine if this is a system resources
issue, and if so, which system resources. is this sequence something
that can be prevented in the future via postgresql.conf? more memory?
is there any way of knowing what actually brought down postgres from
these messages?
this installation is on a Linux box running kernel 2.4.18 with 1GB RAM.
i can provide postgresql.conf settings upon request.
thanks!
-tfo