Re: Postgres database is not logging any messages after the reboot

Here are the messages from the pg_log before and after server reboot.

2007-10-10 06:22:50 CDT  LOG:  archived transaction log file
"000000010000007000000090"
 2007-10-10 06:29:04 CDT  LOG:  autovacuum: processing database "nsc"
 2007-10-10 06:34:58 CDT  LOG:  archived transaction log file
"000000010000007000000091"
 2007-10-10 07:11:48 CDT  LOG:  database system was interrupted at
2007-10-10 06:40:05 CDT
 2007-10-10 07:11:48 CDT  LOG:  checkpoint record is at 70/925E7B68
 2007-10-10 07:11:48 CDT  LOG:  redo record is at 70/925E7B68; undo
record is at 0/0; shutdown FALSE
 2007-10-10 07:11:48 CDT  LOG:  next transaction ID: 656073787; next
OID: 2915367
 2007-10-10 07:11:48 CDT  LOG:  next MultiXactId: 3599; next
MultiXactOffset: 7197
 2007-10-10 07:11:48 CDT  LOG:  database system was not properly shut
down; automatic recovery in progress
 2007-10-10 07:11:48 CDT  LOG:  redo starts at 70/925E7BB8
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
:
:
:
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  record with zero length at 70/92BBF1C8
 2007-10-10 07:11:53 CDT  LOG:  redo done at 70/92BBF198
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  database system is ready
 2007-10-10 07:11:53 CDT  LOG:  transaction ID wrap limit is 2147484146,
limited by database "postgres"
 2007-10-10 07:11:59 CDT  LOG:  archived transaction log file
"000000010000007000000092"
nsc 2007-10-10 07:19:37 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:20:48 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:23:53 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection

Thanks
Paramjeet Kaur

-----Original Message-----
From: Bhella Paramjeet-PFCW67
Sent: Wednesday, October 10, 2007 10:43 AM
To: 'pgsql-admin@postgresql.org'
Cc: Subbiah Stalin-XCGF84
Subject: Postgres database is not logging any messages after the reboot

Hi

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10
06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10
06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox
kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel:
EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02
sandbox kernel: __journal_remove_journal_head: freeing b_committed_data

Thanks
Paramjeet Bhella