Data corruption

We appear to have had some corruption on a customer’s postgres cluster.

They are on 9.0.17 32bit

Windows Server 2003 – Service pack 2

Intel Xeon 2.66GHZ

4GB Memory

Raid is setup but doesn’t look good – just now showing status of Degraded!!

The RAID doesn’t look too well…. currently has status Degraded and on the Segments tab and it’s showing Segment 1 (Missing)

I guess we can assume the issue is down to hardware...

An engineer has been dispatched to replace the hardware and we are arranging to have the cluster shutdown and backed up to a separate storage device.

Their postgresql.conf file is pretty much as it comes with only the following line added to the end:

                custom_variable_classes = 'user_vars'

Everything was fine until 13:28 on 7^th November when there was a number of these entries in the log:

2014-11-07 13:28:45 GMT WARNING:  worker took too long to start; cancelled

After that the log file was cycled and it started with:

2014-11-07 14:15:19 GMT FATAL:  the database system is starting up

2014-11-07 14:15:20 GMT FATAL:  the database system is starting up

2014-11-07 14:15:20 GMT LOG:  database system was interrupted; last known up at 2014-11-07 13:28:42 GMT

2014-11-07 14:15:21 GMT FATAL:  the database system is starting up

2014-11-07 14:15:22 GMT FATAL:  the database system is starting up

2014-11-07 14:15:23 GMT FATAL:  the database system is starting up

2014-11-07 14:15:23 GMT LOG:  database system was not properly shut down; automatic recovery in progress

2014-11-07 14:15:23 GMT LOG:  record with zero length at 5/7B4CAC0

2014-11-07 14:15:23 GMT LOG:  redo is not required

2014-11-07 14:15:24 GMT FATAL:  the database system is starting up

2014-11-07 14:15:25 GMT FATAL:  the database system is starting up

2014-11-07 14:15:25 GMT LOG:  database system is ready to accept connections

2014-11-07 14:15:25 GMT LOG:  autovacuum launcher started

2014-11-07 14:15:33 GMT LOG:  unexpected EOF on client connection

Since then whenever trying to write to or query one particular table we receive the following:

                2014-11-07 15:13:57 GMT ERROR:  invalid page header in block 29838 of relation base/16392/640564

It’s always the same error (block and relation) as far as I can tell.

So the question is, what next?

We may have lost data as it couldn’t be written but it’s not the end of the world.

The more important bit would be to stop any further data loss.

Regards,

Russell Keane

INPS

Tel:         +44 (0)20 7501 7277

Follow us on twitter | visit www.inps.co.uk