I have a 7.0.3 installation, compiled from source, on a (pretty much)
stock RedHat 6.2 system with security patches. There are about 8
databases, none of which are very active. I'm running a web
application (OpenACS 3.2.4) that runs under AOLServer and uses the
databases.
The problem I'm having is that the backends will crash randomly, after
the database has been up for a few days, with:
FATAL 1: Memory exhausted in AllocSetAlloc()
Sometimes this happens while the nightly vacuum is running, sometimes
at random times when the web app makes a query. Here's an example of
the full error from the vacuum script:
Vacuuming template1
VACUUM
Vacuuming nsaweb
VACUUM
Vacuuming wb
VACUUM
Vacuuming eq
VACUUM
Vacuuming penntex
FATAL 1: Memory exhausted in AllocSetAlloc()
pqReadData() -- backend closed the channel unexpectedly.
This probably means the backend terminated abnormally
before or while processing the request.
connection to server was lost
I have core dumps enabled (ulimit -c unlimited) and there are no core
files in the $PGDATA tree. This makes sense, as I looked at the
source, and this message is only generated when malloc() fails, on
which the backend terminates--there's no signal generated.
I've check all other ulimit settings and can't see any limits that PG
would be hitting. I'm running the postmaster as follows:
/usr/local/pgsql/bin/postmaster -N 64 -B 2000 -o "-S 2000" -D $PGDATA
The system has plenty of memory and swap, and under normal
circumstances the backends take up 10-15 megabytes. If it's a
runaway situation of some kind, it happens very fast, as I've even
taken snapshots of the process table at 1 minute intervals, and they
show no abnormality right up to the time of the crash.
Any ideas on what might be the problem? It seems odd that I'm the
only person seeing this level of instability...
-Doug