Scott - thanks, in pouring over the logs, I have not found anything certain, but have turned up a ton of mesages about my sysadmins se-linux security and php/pg (don't know if they're my app or script kiddies...). I will keep looking for things pertianing to these crashes. The message relative to php and pgsql:
Sep 20 11:37:32 deq1 setroubleshoot: SELinux is preventing php (httpd_sys_script_t) "setopt" to <Unknown> (httpd_sys_script_t). For complete SELinux messages. run sealert -l 199268c4-b84d-4a33-a073-29bd4461f875
Joe - it appears that it ALWAYS involves pLR - even a simple median call has caused it, though I must say it is something that is calculating the median of somewhere around 10-20,000 pieces of data if that makes any difference. I would be delighted to run any kind of debugging necessary and share the info. I have an identical system that can reproduce the errors (I am pretty certain that they HAVE previously). What I DON'T have is any knowledge of the stack-trace/debugger things, but I'm willing to learn, and I have a sysadmin who may be able to lend a hand.
Thanks a bunch gents - I have been nibbling around the edges of this problem for quite some time and I am ready to take a bite,
r.b.
-----Original Message-----
From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
Sent: Fri 9/23/2011 3:42 PM
To: Burgholzer, Robert (DEQ)
Cc: Joe Conway; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] diagnosing a db crash - server exit code 2
On Fri, Sep 23, 2011 at 1:38 PM, Burgholzer, Robert (DEQ)
<Robert.Burgholzer@deq.virginia.gov> wrote:
> Joe,
> Thanks - I will try to check into this - however, we have done some tuning
> on the memory over the last 2 years and gotten it such that it is seldom if
> every having to dip into its swap too substantially -- according to "top",
> we remain under 1% swap usage during most times. Generally speaking, I run
> 3-5 of these large PHP processes simultaneously with no issue, however, when
> I issue even a "median" function call, after several calls (no consistent
> pattern that I can discern), the backend crashes.
>
> If this were the OOM killer - any way I would diagnose it?
If it's the OOM killer it'll be in your /var/log/messages log.