Обсуждение: PostgreSQL database and other daemons crashing

I have a Ruby on Rails web site that is running on a CentOS 5 server using Apache 2 with Passenger and PostgreSQL 8.4.  Over the past two weeks I am seeing my PostgreSQL daemon, postfix, and rsyslog daemons all die at the same time, this happens at least 5 times per day at random times.  Apache however, does not die. 

I cannot associate it with any specific URL on the site or database query, it doesn’t match up with any scheduled tasks, and hardly reports anything to most logs.  The only relevant log entry I can find is below from my PostgreSQL logs.  I’ve searched through many PostgreSQL list archives and found little information.  Only the suggestion that something might be sending SIGTERM commands to PostgreSQL or that system limits may be reached.  The server has plenty of free disk space and 1 GB of ram with only 250MB in use.    It’s a dedicated web server with no other processes running really, just the base CentOS processes and the ones I mentioned at the start of this post. 

It seems to be the result of database query within my Ruby code.  The processes do not die when apache isn’t running, but I cannot pin point it.  I’ve even rebuilt the server from scratch and am having the same results.  Has anyone seen anything like this before?

2010-05-19 14:40:45 MST | 127.0.0.1FATAL:  terminating connection due to administrator command
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG:  shmem_exit(1): 6 callbacks to make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG:  proc_exit(1): 3 callbacks to make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG:  exit(1)
2010-05-19 14:40:45 MST | 127.0.0.1LOG:  could not send data to client: Bad file descriptor
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG:  shmem_exit(-1): 0 callbacks to make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG:  proc_exit(-1): 0 callbacks to make

Thanks for any help,
SpencerPostgreSQL database and other daemons crashing

On Thu, 2010-05-20 at 15:10 -0500, Spencer Anderson wrote:
> I have a Ruby on Rails web site that is running on a CentOS 5 server
> using Apache 2 with Passenger and PostgreSQL 8.4.  Over the past two
> weeks I am seeing my PostgreSQL daemon, postfix, and rsyslog daemons
> all die at the same time, this happens at least 5 times per day at
> random times.  Apache however, does not die.

All three? What does /var/log/messages    or /var/log/syslog say?


I wonder if the oom killer is acting up.


Joshua D. Drake


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering

/var/log/messages doesn't really show anything (see below).  Just my repeated manual restarts because of the daemon crashing.  Yes, all three crash each time, I can't figure out why rsyslog and postfix die because of a database issue.  I'm not familiar with the oom killer.  I'll google around.  Do you have any advice on how to test that as a cause?

May 20 09:59:28 kernel: rklogd 2.0.6, log source = /proc/kmsg started.
May 20 10:22:51 rsyslogd: [origin software="rsyslogd" swVersion="2.0.6" x-pid="18289" x-info="http://www.rsyslog.com"][x-configInfo udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
May 20 10:22:51 kernel: rklogd 2.0.6, log source = /proc/kmsg started.
May 20 11:26:26 rsyslogd: [origin software="rsyslogd" swVersion="2.0.6" x-pid="17507" x-info="http://www.rsyslog.com"][x-configInfo udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart





--
Spencer Anderson
spencer@minneapp.com
(424) 901-1363

On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> I wonder if the oom killer is acting up.
>

that'd be my guess.  postfix *never* dies.  something external is
killing your processes.

That may in fact be the cause.  I ran,  free -lm -s 0.5, to watch my memory use and saw the output below.  My processes didn't die this time when it hit zero bytes free, but it only lasted for 1 second.  Some process must be occasionally using very excessive memory, and every once in a while, I get the OOM kill scenario.    Guess I need to find that process..

             total       used       free     shared    buffers     cached
Mem:          1024        176        847          0          0          0
Low:          1024        176        847
High:            0          0          0
-/+ buffers/cache:        176        847
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:          1024        176        847          0          0          0
Low:          1024        176        847
High:            0          0          0
-/+ buffers/cache:        176        847
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:          1024       1024          0          0          0          0
Low:          1024       1024          0
High:            0          0          0
-/+ buffers/cache:       1024          0
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:          1024       1024          0          0          0          0
Low:          1024       1024          0
High:            0          0          0
-/+ buffers/cache:       1024          0
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:          1024        176        847          0          0          0
Low:          1024        176        847
High:            0          0          0
-/+ buffers/cache:        176        847
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:          1024        176        847          0          0          0
Low:          1024        176        847
High:            0          0          0
-/+ buffers/cache:        176        847
Swap:            0          0          0
 




--
Spencer Anderson
spencer@minneapp.com
(424) 901-1363

Vick Khera <vivek@khera.org> writes:
> On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
>> I wonder if the oom killer is acting up.

> that'd be my guess.  postfix *never* dies.  something external is
> killing your processes.

Either that or his hardware is failing, eg RAM dropping critical bits in
the memory of several processes at the same time.

            regards, tom lane

On Thu, May 20, 2010 at 3:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Vick Khera <vivek@khera.org> writes:
>> On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
>>> I wonder if the oom killer is acting up.
>
>> that'd be my guess.  postfix *never* dies.  something external is
>> killing your processes.
>
> Either that or his hardware is failing, eg RAM dropping critical bits in
> the memory of several processes at the same time.

Defintely worth cranking up an afternoon of memtest86+ to see.

On Thu, 2010-05-20 at 15:10 -0500, Spencer Anderson wrote:
> I have a Ruby on Rails web site that is running on a CentOS 5 server
> using Apache 2 with Passenger and PostgreSQL 8.4.  Over the past two
> weeks I am seeing my PostgreSQL daemon, postfix, and rsyslog daemons
> all die at the same time, this happens at least 5 times per day at
> random times.  Apache however, does not die.

All three? What does /var/log/messages    or /var/log/syslog say?


I wonder if the oom killer is acting up.


Joshua D. Drake


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering