Обсуждение: Database shut down unexpectedly.
Today the database shut down unexpectedly. I have included the log file that shows the shutdown. Can anybody tell me why this happened and how I can make sure it doesn't happen again.
The only thing I can think of that I did was to specify a password for the postgres user in the operating system.
Here is the log file. Very strange.
2009-03-25 00:02:01 GMT LOG: incomplete startup packet
2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
2009-03-25 00:32:15 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:32:15 GMT LOG: unexpected EOF on client connection
2009-03-25 02:41:57 GMT LOG: incomplete startup packet
2009-03-25 02:41:57 GMT LOG: received smart shutdown request
2009-03-25 02:41:57 GMT LOG: autovacuum launcher shutting down
2009-03-25 02:45:03 GMT FATAL: the database system is shutting down
2009-03-25 02:45:03 GMT FATAL: the database system is shutting down
2009-03-25 02:50:02 GMT FATAL: the database system is shutting down
2009-03-25 02:50:02 GMT FATAL: the database system is shutting down
2009-03-25 02:55:03 GMT FATAL: the database system is shutting down
2009-03-25 02:55:03 GMT FATAL: the database system is shutting down
2009-03-25 03:00:03 GMT FATAL: the database system is shutting down
2009-03-25 03:00:03 GMT FATAL: the database system is shutting down
2009-03-25 03:05:03 GMT FATAL: the database system is shutting down
2009-03-25 03:05:03 GMT FATAL: the database system is shutting down
2009-03-25 03:10:02 GMT FATAL: the database system is shutting down
2009-03-25 03:10:02 GMT FATAL: the database system is shutting down
2009-03-25 03:15:03 GMT FATAL: the database system is shutting down
2009-03-25 03:15:03 GMT FATAL: the database system is shutting down
2009-03-25 03:20:03 GMT FATAL: the database system is shutting down
2009-03-25 03:20:03 GMT FATAL: the database system is shutting down
2009-03-25 03:25:02 GMT FATAL: the database system is shutting down
2009-03-25 03:25:02 GMT FATAL: the database system is shutting down
2009-03-25 03:30:03 GMT FATAL: the database system is shutting down
2009-03-25 03:30:03 GMT FATAL: the database system is shutting down
2009-03-25 03:35:03 GMT FATAL: the database system is shutting down
2009-03-25 03:35:03 GMT FATAL: the database system is shutting down
2009-03-25 03:40:02 GMT FATAL: the database system is shutting down
2009-03-25 03:40:02 GMT FATAL: the database system is shutting down
2009-03-25 03:45:02 GMT FATAL: the database system is shutting down
2009-03-25 03:45:02 GMT FATAL: the database system is shutting down
2009-03-25 03:50:03 GMT FATAL: the database system is shutting down
2009-03-25 03:50:03 GMT FATAL: the database system is shutting down
2009-03-25 03:55:03 GMT FATAL: the database system is shutting down
2009-03-25 03:55:03 GMT FATAL: the database system is shutting down
2009-03-25 04:00:02 GMT FATAL: the database system is shutting down
2009-03-25 04:00:02 GMT FATAL: the database system is shutting down
2009-03-25 04:25:03 GMT FATAL: the database system is shutting down
2009-03-25 04:30:03 GMT FATAL: the database system is shutting down
2009-03-25 04:30:03 GMT FATAL: the database system is shutting down
2009-03-25 04:35:02 GMT FATAL: the database system is shutting down
2009-03-25 04:35:02 GMT FATAL: the database system is shutting down
2009-03-25 04:40:03 GMT FATAL: the database system is shutting down
2009-03-25 04:40:03 GMT FATAL: the database system is shutting down
2009-03-25 04:45:03 GMT FATAL: the database system is shutting down
2009-03-25 04:45:03 GMT FATAL: the database system is shutting down
2009-03-25 04:50:02 GMT FATAL: the database system is shutting down
2009-03-25 04:50:02 GMT FATAL: the database system is shutting down
2009-03-25 04:55:03 GMT FATAL: the database system is shutting down
2009-03-25 04:55:03 GMT FATAL: the database system is shutting down
2009-03-25 05:00:03 GMT FATAL: the database system is shutting down
2009-03-25 05:00:03 GMT FATAL: the database system is shutting down
2009-03-25 05:02:01 GMT LOG: incomplete startup packet
2009-03-25 05:05:02 GMT FATAL: the database system is shutting down
2009-03-25 05:05:02 GMT FATAL: the database system is shutting down
2009-03-25 05:10:02 GMT FATAL: the database system is shutting down
2009-03-25 05:10:02 GMT FATAL: the database system is shutting down
2009-03-25 05:15:03 GMT FATAL: the database system is shutting down
2009-03-25 05:15:03 GMT FATAL: the database system is shutting down
2009-03-25 05:20:03 GMT FATAL: the database system is shutting down
2009-03-25 05:20:03 GMT FATAL: the database system is shutting down
2009-03-25 05:25:02 GMT FATAL: the database system is shutting down
2009-03-25 05:25:02 GMT FATAL: the database system is shutting down
2009-03-25 05:30:03 GMT FATAL: the database system is shutting down
2009-03-25 05:30:03 GMT FATAL: the database system is shutting down
2009-03-25 05:35:03 GMT FATAL: the database system is shutting down
2009-03-25 05:35:03 GMT FATAL: the database system is shutting down
2009-03-25 05:40:02 GMT FATAL: the database system is shutting down
2009-03-25 05:40:02 GMT FATAL: the database system is shutting down
2009-03-25 05:45:02 GMT FATAL: the database system is shutting down
2009-03-25 05:45:02 GMT FATAL: the database system is shutting down
2009-03-25 05:50:03 GMT FATAL: the database system is shutting down
2009-03-25 05:50:03 GMT FATAL: the database system is shutting down
2009-03-25 05:55:03 GMT FATAL: the database system is shutting down
2009-03-25 05:55:03 GMT FATAL: the database system is shutting down
2009-03-25 06:00:02 GMT FATAL: the database system is shutting down
2009-03-25 06:00:02 GMT FATAL: the database system is shutting down
2009-03-25 06:05:03 GMT FATAL: the database system is shutting down
2009-03-25 06:05:03 GMT FATAL: the database system is shutting down
2009-03-25 06:10:03 GMT FATAL: the database system is shutting down
2009-03-25 06:10:03 GMT FATAL: the database system is shutting down
2009-03-25 06:15:02 GMT FATAL: the database system is shutting down
2009-03-25 06:15:02 GMT FATAL: the database system is shutting down
2009-03-25 06:20:03 GMT FATAL: the database system is shutting down
2009-03-25 06:20:03 GMT FATAL: the database system is shutting down
2009-03-25 06:25:03 GMT FATAL: the database system is shutting down
2009-03-25 06:25:03 GMT FATAL: the database system is shutting down
2009-03-25 06:30:03 GMT FATAL: the database system is shutting down
2009-03-25 06:30:03 GMT FATAL: the database system is shutting down
2009-03-25 06:35:02 GMT FATAL: the database system is shutting down
2009-03-25 06:35:02 GMT FATAL: the database system is shutting down
2009-03-25 06:40:03 GMT FATAL: the database system is shutting down
2009-03-25 06:40:03 GMT FATAL: the database system is shutting down
2009-03-25 06:45:03 GMT FATAL: the database system is shutting down
2009-03-25 06:45:03 GMT FATAL: the database system is shutting down
2009-03-25 06:50:02 GMT FATAL: the database system is shutting down
2009-03-25 06:50:02 GMT FATAL: the database system is shutting down
2009-03-25 06:55:03 GMT FATAL: the database system is shutting down
2009-03-25 06:55:03 GMT FATAL: the database system is shutting down
2009-03-25 07:00:03 GMT FATAL: the database system is shutting down
2009-03-25 07:00:03 GMT FATAL: the database system is shutting down
2009-03-25 07:05:03 GMT FATAL: the database system is shutting down
2009-03-25 07:05:03 GMT FATAL: the database system is shutting down
2009-03-25 07:06:42 GMT LOG: could not receive data from client: Connection reset by peer
2009-03-25 07:06:42 GMT LOG: unexpected EOF on client connection
2009-03-25 07:06:42 GMT LOG: shutting down
2009-03-25 07:06:42 GMT LOG: database system is shut down
2009-03-25 08:54:47 GMT LOG: could not load root certificate file "root.crt": no SSL error reported
2009-03-25 08:54:47 GMT DETAIL: Will not verify client certificates.
2009-03-25 08:54:47 GMT LOG: database system was shut down at 2009-03-25 07:06:42 GMT
2009-03-25 08:54:47 GMT LOG: autovacuum launcher started
2009-03-25 08:54:47 GMT LOG: database system is ready to accept connections
2009-03-25 08:54:48 GMT LOG: incomplete startup packet
The only thing I can think of that I did was to specify a password for the postgres user in the operating system.
Here is the log file. Very strange.
2009-03-25 00:02:01 GMT LOG: incomplete startup packet
2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
2009-03-25 00:32:15 GMT LOG: could not receive data from client: Connection timed out
2009-03-25 00:32:15 GMT LOG: unexpected EOF on client connection
2009-03-25 02:41:57 GMT LOG: incomplete startup packet
2009-03-25 02:41:57 GMT LOG: received smart shutdown request
2009-03-25 02:41:57 GMT LOG: autovacuum launcher shutting down
2009-03-25 02:45:03 GMT FATAL: the database system is shutting down
2009-03-25 02:45:03 GMT FATAL: the database system is shutting down
2009-03-25 02:50:02 GMT FATAL: the database system is shutting down
2009-03-25 02:50:02 GMT FATAL: the database system is shutting down
2009-03-25 02:55:03 GMT FATAL: the database system is shutting down
2009-03-25 02:55:03 GMT FATAL: the database system is shutting down
2009-03-25 03:00:03 GMT FATAL: the database system is shutting down
2009-03-25 03:00:03 GMT FATAL: the database system is shutting down
2009-03-25 03:05:03 GMT FATAL: the database system is shutting down
2009-03-25 03:05:03 GMT FATAL: the database system is shutting down
2009-03-25 03:10:02 GMT FATAL: the database system is shutting down
2009-03-25 03:10:02 GMT FATAL: the database system is shutting down
2009-03-25 03:15:03 GMT FATAL: the database system is shutting down
2009-03-25 03:15:03 GMT FATAL: the database system is shutting down
2009-03-25 03:20:03 GMT FATAL: the database system is shutting down
2009-03-25 03:20:03 GMT FATAL: the database system is shutting down
2009-03-25 03:25:02 GMT FATAL: the database system is shutting down
2009-03-25 03:25:02 GMT FATAL: the database system is shutting down
2009-03-25 03:30:03 GMT FATAL: the database system is shutting down
2009-03-25 03:30:03 GMT FATAL: the database system is shutting down
2009-03-25 03:35:03 GMT FATAL: the database system is shutting down
2009-03-25 03:35:03 GMT FATAL: the database system is shutting down
2009-03-25 03:40:02 GMT FATAL: the database system is shutting down
2009-03-25 03:40:02 GMT FATAL: the database system is shutting down
2009-03-25 03:45:02 GMT FATAL: the database system is shutting down
2009-03-25 03:45:02 GMT FATAL: the database system is shutting down
2009-03-25 03:50:03 GMT FATAL: the database system is shutting down
2009-03-25 03:50:03 GMT FATAL: the database system is shutting down
2009-03-25 03:55:03 GMT FATAL: the database system is shutting down
2009-03-25 03:55:03 GMT FATAL: the database system is shutting down
2009-03-25 04:00:02 GMT FATAL: the database system is shutting down
2009-03-25 04:00:02 GMT FATAL: the database system is shutting down
2009-03-25 04:25:03 GMT FATAL: the database system is shutting down
2009-03-25 04:30:03 GMT FATAL: the database system is shutting down
2009-03-25 04:30:03 GMT FATAL: the database system is shutting down
2009-03-25 04:35:02 GMT FATAL: the database system is shutting down
2009-03-25 04:35:02 GMT FATAL: the database system is shutting down
2009-03-25 04:40:03 GMT FATAL: the database system is shutting down
2009-03-25 04:40:03 GMT FATAL: the database system is shutting down
2009-03-25 04:45:03 GMT FATAL: the database system is shutting down
2009-03-25 04:45:03 GMT FATAL: the database system is shutting down
2009-03-25 04:50:02 GMT FATAL: the database system is shutting down
2009-03-25 04:50:02 GMT FATAL: the database system is shutting down
2009-03-25 04:55:03 GMT FATAL: the database system is shutting down
2009-03-25 04:55:03 GMT FATAL: the database system is shutting down
2009-03-25 05:00:03 GMT FATAL: the database system is shutting down
2009-03-25 05:00:03 GMT FATAL: the database system is shutting down
2009-03-25 05:02:01 GMT LOG: incomplete startup packet
2009-03-25 05:05:02 GMT FATAL: the database system is shutting down
2009-03-25 05:05:02 GMT FATAL: the database system is shutting down
2009-03-25 05:10:02 GMT FATAL: the database system is shutting down
2009-03-25 05:10:02 GMT FATAL: the database system is shutting down
2009-03-25 05:15:03 GMT FATAL: the database system is shutting down
2009-03-25 05:15:03 GMT FATAL: the database system is shutting down
2009-03-25 05:20:03 GMT FATAL: the database system is shutting down
2009-03-25 05:20:03 GMT FATAL: the database system is shutting down
2009-03-25 05:25:02 GMT FATAL: the database system is shutting down
2009-03-25 05:25:02 GMT FATAL: the database system is shutting down
2009-03-25 05:30:03 GMT FATAL: the database system is shutting down
2009-03-25 05:30:03 GMT FATAL: the database system is shutting down
2009-03-25 05:35:03 GMT FATAL: the database system is shutting down
2009-03-25 05:35:03 GMT FATAL: the database system is shutting down
2009-03-25 05:40:02 GMT FATAL: the database system is shutting down
2009-03-25 05:40:02 GMT FATAL: the database system is shutting down
2009-03-25 05:45:02 GMT FATAL: the database system is shutting down
2009-03-25 05:45:02 GMT FATAL: the database system is shutting down
2009-03-25 05:50:03 GMT FATAL: the database system is shutting down
2009-03-25 05:50:03 GMT FATAL: the database system is shutting down
2009-03-25 05:55:03 GMT FATAL: the database system is shutting down
2009-03-25 05:55:03 GMT FATAL: the database system is shutting down
2009-03-25 06:00:02 GMT FATAL: the database system is shutting down
2009-03-25 06:00:02 GMT FATAL: the database system is shutting down
2009-03-25 06:05:03 GMT FATAL: the database system is shutting down
2009-03-25 06:05:03 GMT FATAL: the database system is shutting down
2009-03-25 06:10:03 GMT FATAL: the database system is shutting down
2009-03-25 06:10:03 GMT FATAL: the database system is shutting down
2009-03-25 06:15:02 GMT FATAL: the database system is shutting down
2009-03-25 06:15:02 GMT FATAL: the database system is shutting down
2009-03-25 06:20:03 GMT FATAL: the database system is shutting down
2009-03-25 06:20:03 GMT FATAL: the database system is shutting down
2009-03-25 06:25:03 GMT FATAL: the database system is shutting down
2009-03-25 06:25:03 GMT FATAL: the database system is shutting down
2009-03-25 06:30:03 GMT FATAL: the database system is shutting down
2009-03-25 06:30:03 GMT FATAL: the database system is shutting down
2009-03-25 06:35:02 GMT FATAL: the database system is shutting down
2009-03-25 06:35:02 GMT FATAL: the database system is shutting down
2009-03-25 06:40:03 GMT FATAL: the database system is shutting down
2009-03-25 06:40:03 GMT FATAL: the database system is shutting down
2009-03-25 06:45:03 GMT FATAL: the database system is shutting down
2009-03-25 06:45:03 GMT FATAL: the database system is shutting down
2009-03-25 06:50:02 GMT FATAL: the database system is shutting down
2009-03-25 06:50:02 GMT FATAL: the database system is shutting down
2009-03-25 06:55:03 GMT FATAL: the database system is shutting down
2009-03-25 06:55:03 GMT FATAL: the database system is shutting down
2009-03-25 07:00:03 GMT FATAL: the database system is shutting down
2009-03-25 07:00:03 GMT FATAL: the database system is shutting down
2009-03-25 07:05:03 GMT FATAL: the database system is shutting down
2009-03-25 07:05:03 GMT FATAL: the database system is shutting down
2009-03-25 07:06:42 GMT LOG: could not receive data from client: Connection reset by peer
2009-03-25 07:06:42 GMT LOG: unexpected EOF on client connection
2009-03-25 07:06:42 GMT LOG: shutting down
2009-03-25 07:06:42 GMT LOG: database system is shut down
2009-03-25 08:54:47 GMT LOG: could not load root certificate file "root.crt": no SSL error reported
2009-03-25 08:54:47 GMT DETAIL: Will not verify client certificates.
2009-03-25 08:54:47 GMT LOG: database system was shut down at 2009-03-25 07:06:42 GMT
2009-03-25 08:54:47 GMT LOG: autovacuum launcher started
2009-03-25 08:54:47 GMT LOG: database system is ready to accept connections
2009-03-25 08:54:48 GMT LOG: incomplete startup packet
In response to Tim Uckun <timuckun@gmail.com>:
> Today the database shut down unexpectedly. I have included the log file
> that shows the shutdown. Can anybody tell me why this happened and how I can
> make sure it doesn't happen again.
>
> The only thing I can think of that I did was to specify a password for the
> postgres user in the operating system.
Not likely to cause the DB to restart ... at least not in any OS
configuration that I'm aware of. However, you don't mention what
OS you're running ... that might be important.
> Here is the log file. Very strange.
>
> 2009-03-25 00:02:01 GMT LOG: incomplete startup packet
> 2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
> 2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
> 2009-03-25 00:32:15 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:32:15 GMT LOG: unexpected EOF on client connection
> 2009-03-25 02:41:57 GMT LOG: incomplete startup packet
> 2009-03-25 02:41:57 GMT LOG: received smart shutdown request
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sure looks like someone intentionally shut the database down.
Who has login access to this system with enough privs to do that? I'd
start by questioning them on their activities at that time. If that
doesn't answer your question, then implement OS auditing so you can
catch the culprit next time. It's quite possible that someone is sending
signals to the DB system without knowledge of how those signals are
interpreted.
--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/
On Thu, Mar 26, 2009 at 2:23 AM, Bill Moran <wmoran@potentialtech.com> wrote:
That was most likely me (doing a /etc/init.d/postgresql restart. I thought I would restart it after changing the user name. Notice that the time on that is 02:41 GMT. The actual shutdown occured on 07:05 GMT some four hours later.
Does that make sense?
In response to Tim Uckun <timuckun@gmail.com>:Not likely to cause the DB to restart ... at least not in any OS
> Today the database shut down unexpectedly. I have included the log file
> that shows the shutdown. Can anybody tell me why this happened and how I can
> make sure it doesn't happen again.
>
> The only thing I can think of that I did was to specify a password for the
> postgres user in the operating system.
configuration that I'm aware of. However, you don't mention what
OS you're running ... that might be important.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Here is the log file. Very strange.
>
> 2009-03-25 00:02:01 GMT LOG: incomplete startup packet
> 2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
> 2009-03-25 00:30:05 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:30:05 GMT LOG: unexpected EOF on client connection
> 2009-03-25 00:32:15 GMT LOG: could not receive data from client: Connection
> timed out
> 2009-03-25 00:32:15 GMT LOG: unexpected EOF on client connection
> 2009-03-25 02:41:57 GMT LOG: incomplete startup packet
> 2009-03-25 02:41:57 GMT LOG: received smart shutdown request
Sure looks like someone intentionally shut the database down.
That was most likely me (doing a /etc/init.d/postgresql restart. I thought I would restart it after changing the user name. Notice that the time on that is 02:41 GMT. The actual shutdown occured on 07:05 GMT some four hours later.
Does that make sense?