I've got a database that reported this error yesterday morning after
an UPDATE statement:
ERROR: invalid page header in block 34 of relation "pg_toast_167245"
Later in the day, it reported this one:
ERROR: invalid page header in block 199 of relation "<table1>_pkey"
this time after a SELECT.
Before these errors had been brought to my attention, I was looking
late that night at a long-running query on an unrelated table. There
was a query that should've finished in seconds that was taking hours.
I later heard from a developer that there were some deadlock-related
issues with the query in question, and the development team wound up
issuing:
pg_ctl kill QUIT [process_id]
to get rid of the problem query this morning, which was a relatively
unimportant SELECT.
Then, shortly thereafter, they began seeing yet another invalid page
header:
invalid page header in block 4369 of relation "<table2>"
So there are currently three separate relations exhibiting invalid
page errors.
This box is a Debian 3.1 box running a custom Linux 2.6.10 #6 SMP
kernel. Postgres 8.1.3 was compiled from source. pgpool 3.0.1, also
built from source, is used by some parts of the application layer.
The system is running on an ext3 filesystem, WAL is on a 4-disk RAID
10 running JFS, and data is on a 12-disk RAID 10 running JFS. I'm not
seeing any signs of apparent kernel or hardware errors in the system
and kernel logs.
Also see nearby thread of a troubling error from the night before the
first sight of invalid page headers:
http://archives.postgresql.org/pgsql-general/2006-04/msg00746.php
Is there any way in which this could be related to the invalid page
headers?
Based on this thread:
http://archives.postgresql.org/pgsql-general/2005-11/msg01140.php
I'm a little nervous about the prospects for analysis and recovery
here. Any thoughts?
Is there a risk that if we took postgres offline in this state that
it would not come back up?
--
Thomas F. O'Connell
Database Architecture and Programming
Sitening, LLC
http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)