Обсуждение: invalid page header in block 29 of relation "pg_type"
Hello, we have (maybe due to temporary Disk Array HW failure, which occurred on the same day the pg_dump started to complain - array is now successfully rebuilt) problem with consistency of pg_catalog.pg_type table. pg_dumpall command gives following message: pg_dump: SQL command failed pg_dump: Error message from server: ERROR: invalid page header in block 29 of relation "pg_type" pg_dump: The command was: SELECT tableoid, oid, typname, typnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = typowner) as rolname, typinput::oid as typinput, typoutput::oid as typoutput, typelem, typrelid, CASE WHEN typrelid = 0 THEN ' '::"char" ELSE (SELECT relkind FROM pg_class WHERE oid = typrelid) END as typrelkind, typtype, typisdefined FROM pg_type pg_dumpall: pg_dump failed on database "isdb", exiting and exits, so we are not able to make fresh backup and recover the database. The same error (ERROR: invalid page header in block 29 of relation "pg_type") give commands VACUUM, REINDEX on the table pg_catalog.pg_type. I'm afraid of performing any tests, restarting the service, because it is running production database and due to the inability of backing it up we have only week old dump. The service acts as there was no problem (INSERT, SELECT, DROP, CREATE, ALTER TABLE works normally), only the dump and vacuum complains. There was one damaged file (fortunetely belonging to an empty regular table) in the $PGDATA/base/ directory, which caused I/O error, while reading. After drop and create of the table, the partition with $PGDATA seems error-free. We run PostgreSQL 8.1.5 on CentOS release 3.8, 2.4.21-47.EL kernel, x86_64 architecture. Thanks for any advice Best regards, Filip Krska -- ComSTAR spol. s r. o. Třebohostická 14 100 31 Praha 10 HotLine (pev.): 261 305 432 HotLine (mob.): 777 343 857
Filip Krška wrote: > Hello, > > we have (maybe due to temporary Disk Array HW failure, which occurred on > the same day the pg_dump started to complain - array is now successfully > rebuilt) problem with consistency of pg_catalog.pg_type table. Did you solve this problem? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
I hope, I've solved it. I found 512 damaged bytes on the 29th 8k page of the pg_type's datafile ($PGDATA/base/16390/1247), it was overwritten by text from some log(?!). When I had dropped the whole 8k page (dd of=/tmp/head count=29 bs=8192, dd of=/tmp/tail skip=30 bs=8192, then cat /tmp/head /tmp/tail), the pg_dumpall utility was able to dump the DB instance then. I compared schemas (pg_dumpall -s), fortunately only several empty, not used tables were affected. Then I imported the dump, and now the DB instance seems to work properly, also the pg_dumps, vacuum anylysis. Best regards, Filip Krska -- ComSTAR spol. s r. o. Třebohostická 14 100 31 Praha 10 HotLine (pev.): 274 016 000 HotLine (mob.): 777 343 857 On Fri, 6 Jun 2008, Alvaro Herrera wrote: > Filip Krška wrote: >> Hello, >> >> we have (maybe due to temporary Disk Array HW failure, which occurred on >> the same day the pg_dump started to complain - array is now successfully >> rebuilt) problem with consistency of pg_catalog.pg_type table. > > Did you solve this problem? > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > The PostgreSQL Company - Command Prompt, Inc. > > -- > Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-admin >
Filip Krška wrote: > I hope, I've solved it. > > I found 512 damaged bytes on the 29th 8k page of the pg_type's datafile > ($PGDATA/base/16390/1247), it was overwritten by text from some log(?!). Ugh. For the record, what operating system and filesystem are you using? Oh, the underlying storage may be useful too. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
It was CentOS release 3.8 and ext2 (I switched it to ext3 after the disaster). HW is HP Array P400, with HP DG146ABAB4 Disks. Status of one of the disk switched from OK to predictive failure, I guess the array didn't recognize the failure and didn't park it in time. Filip ====================================== On Mon, 9 Jun 2008, Alvaro Herrera wrote: > Filip Krška wrote: >> I hope, I've solved it. >> >> I found 512 damaged bytes on the 29th 8k page of the pg_type's datafile >> ($PGDATA/base/16390/1247), it was overwritten by text from some log(?!). > > Ugh. For the record, what operating system and filesystem are you > using? Oh, the underlying storage may be useful too. > > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > The PostgreSQL Company - Command Prompt, Inc. > > -- > Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-admin >