Обсуждение: Re: invalid page header

"Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
>
> CETERROR:  XX001: invalid page header in block 22182 of relation
"dunn_main"
>
> My main question is: why is this occuring?
>

I suppose there is no system error happens during the period (like lost
power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
"(PageHeader) bufBlock"?

Regards,
Qingqing

"Qingqing Zhou" <zhouqq@cs.toronto.edu> writes:
> "Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
>> CETERROR:  XX001: invalid page header in block 22182 of relation
> "dunn_main"

> I suppose there is no system error happens during the period (like lost
> power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
> "(PageHeader) bufBlock"?

Probably easier (especially if he's not using a debug-enabled build):
dump out the page from the disk file using pg_filedump, or even just
"dd | od -x".

            regards, tom lane

Tom Lane wrote:
> "Qingqing Zhou" <zhouqq@cs.toronto.edu> writes:
>
>>"Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
>>
>>>CETERROR:  XX001: invalid page header in block 22182 of relation
>>
>>"dunn_main"
>
>
>>I suppose there is no system error happens during the period (like lost
>>power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
>>"(PageHeader) bufBlock"?
>
>
> Probably easier (especially if he's not using a debug-enabled build):
> dump out the page from the disk file using pg_filedump, or even just
> "dd | od -x".
>

Thanks for the 'easier' alternative.  I tried to do this last friday,
but i was amazed when a count(*) on the table and a rerun of the query
that had caused the invalid page header error to occur just worked now.

I asked the developper to delete all imported data again an restart the
import.  This import crashed again with the same error but this time on
another block.

2006-03-27 00:15:25.458 CESTERROR:  XX001: invalid page header in block
48068 of relation "dunn_main"
2006-03-27 00:15:25.458 CESTCONTEXT:  SQL statement "SELECT  phone FROM
dunn_main WHERE source_id =  $1  AND duns =  $2 "
         PL/pgSQL function "proc_dunn" line 29 at select into variables
2006-03-27 00:15:25.458 CESTLOCATION:  ReadBuffer, bufmgr.c:257
2006-03-27 00:15:25.458 CESTSTATEMENT:  SELECT proc_dunn ('J M
Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)

But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)' statement now, it works without errors.

If I would like to dump block 48068 now with pg_dumpfile, how do i know
which file this block belongs to?

Jo.

Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
> I asked the developper to delete all imported data again an restart the
> import.  This import crashed again with the same error but this time on
> another block.

> 2006-03-27 00:15:25.458 CESTERROR:  XX001: invalid page header in block
> 48068 of relation "dunn_main"
> 2006-03-27 00:15:25.458 CESTCONTEXT:  SQL statement "SELECT  phone FROM
> dunn_main WHERE source_id =  $1  AND duns =  $2 "
>          PL/pgSQL function "proc_dunn" line 29 at select into variables
> 2006-03-27 00:15:25.458 CESTLOCATION:  ReadBuffer, bufmgr.c:257
> 2006-03-27 00:15:25.458 CESTSTATEMENT:  SELECT proc_dunn ('J M
> Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
> 54,387166)

> But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
> 3BU','215517942','1','01','S',NULL,'0219',156,1
> 54,387166)' statement now, it works without errors.

That is *really* strange.  Are you certain that the function is
examining the same table you are?  I'm wondering about multiple
similarly-named tables in different schemas, or something like that.

> If I would like to dump block 48068 now with pg_dumpfile, how do i know
> which file this block belongs to?

See
http://www.postgresql.org/docs/8.1/static/storage.html
and/or use contrib/oid2name.

            regards, tom lane

Ok,  So we reran everything and got the same error message again, now
i'm able to reproduce it.

2006-03-28 12:05:18.638 CESTERROR:  XX001: invalid page header in block
39248 of relation "dunn_main"
2006-03-28 12:05:18.638 CESTLOCATION:  ReadBuffer, bufmgr.c:257
2006-03-28 12:05:18.638 CESTSTATEMENT:  VACUUM;

  The output of the block in question is as follows:

* File: /backup/pgsql/data/base/16384/934173
* Options used: -R 39248 -f
*
* Dump created on: Tue Mar 28 13:43:09 2006
*******************************************************************

Block 39248 ********************************************************
<Header> -----
  Block Offset: 0x132a0000         Offsets: Lower     536 (0x0218)
  Block: Size 8192  Version    2            Upper    4796 (0x12bc)
  LSN:  logid     26 recoff 0xa9cad42c      Special  8176 (0x1ff0)
  Items:  129                   Free Space: 4260
  Length (including item array): 540

  Error: Invalid header information.

   0000: 1a000000 2cd4caa9 01000000 1802bc12  ....,...........
   0010: f01f0220 d49f3800 b49f4000 949f4000  ... ..8...@...@.
   0020: 749f4000 5c9f3000 449f3000 2c9f3000  t.@.\.0.D.0.,.0.
   0030: 149f3000 fc9e3000 e49e3000 cc9e3000  ..0...0...0...0.
   0040: b49e3000 9c9e3000 849e3000 6c9e3000  ..0...0...0.l.0.
   0050: 549e3000 3c9e3000 249e3000 0c9e3000  T.0.<.0.$.0...0.
   0060: f49d3000 dc9d3000 c49d3000 ac9d3000  ..0...0...0...0.
   0070: 949d3000 7c9d3000 649d3000 4c9d3000  ..0.|.0.d.0.L.0.
   0080: 289d4800 049d4800 e49c4000 c89c3800  (.H...H...@...8.
   0090: a89c4000 909c3000 6c9c4800 549c3000  ..@...0.l.H.T.0.
   00a0: ec923800 389c3800 1c9c3800 fc9b4000  ..8.8.8...8...@.
   00b0: dc9b4000 bc9b4000 9c9b4000 7c9b4000  ..@...@...@.|.@.
   00c0: 5c9b4000 3c9b4000 1c9b4000 009b3800  \.@.<.@...@...8.
   00d0: e89a3000 d09a3000 b89a3000 a09a3000  ..0...0...0...0.
   00e0: 849a3800 689a3800 509a3000 2c9a4800  ..8.h.8.P.0.,.H.
   00f0: 0c9a4000 bc923000 d4923000 08933000  ..@...0...0...0.
   0100: f4993000 dc993000 c4993000 ac993000  ..0...0...0...0.
   0110: 94993000 7c993000 64993000 4c993000  ..0.|.0.d.0.L.0.
   0120: 34993000 1c993000 04993000 ec983000  4.0...0...0...0.
   0130: d4983000 bc983000 a4983000 8c983000  ..0...0...0...0.
   0140: 74983000 5c983000 44983000 2c983000  t.0.\.0.D.0.,.0.
   0150: 14983000 fc973000 e4973000 cc973000  ..0...0...0...0.
   0160: b4973000 9c973000 84973000 6c973000  ..0...0...0.l.0.
   0170: 54973000 3c973000 24973000 0c973000  T.0.<.0.$.0...0.
   0180: f4963000 dc963000 c4963000 ac963000  ..0...0...0...0.
   0190: 94963000 7c963000 64963000 4c963000  ..0.|.0.d.0.L.0.
   01a0: 34963000 1c963000 04963000 ec953000  4.0...0...0...0.
   01b0: d4953000 bc953000 a4953000 8c953000  ..0...0...0...0.
   01c0: 74953000 5c953000 44953000 2c953000  t.0.\.0.D.0.,.0.
   01d0: 14953000 fc943000 e4943000 cc943000  ..0...0...0...0.
   01e0: b4943000 9c943000 80943800 64943800  ..0...0...8.d.8.
   01f0: 48943800 2c943800 10943800 f4933800  H.8.,.8...8...8.
   0200: cc935000 a8934800 80935000 58935000  ..P...H...P.X.P.
   0210: 3c933800 20933800 20933800           <.8. .8. .8.

<Data> ------
  Item   1 -- Length:   28  Offset: 8148 (0x1fd4)  Flags: USED
   1fd4: 00000f45 20001c40 13000000 62727574  ...E ..@....brut
   1fe4: 6f2d7569 746b6572 696e6700           o-uitkering.

  Item   2 -- Length:   32  Offset: 8116 (0x1fb4)  Flags: USED
   1fb4: 0000dd92 32002040 16000000 62727574  ....2. @....brut
   1fc4: 6f2d6d61 616e6473 616c6172 69730000  o-maandsalaris..

  Item   3 -- Length:   32  Offset: 8084 (0x1f94)  Flags: USED
   1f94: 00008e29 15002040 18000000 62727574  ...).. @....brut
   1fa4: 6f6d6161 6e647361 6c617269 7373656e  omaandsalarissen

  Item   4 -- Length:   32  Offset: 8052 (0x1f74)  Flags: USED
   1f74: 0000af3a 32002040 18000000 62727574  ...:2. @....brut
   1f84: 6f6d6161 6e647361 6c617269 7373656e  omaandsalarissen

  Item   5 -- Length:   24  Offset: 8028 (0x1f5c)  Flags: USED
   1f5c: 0000a70c 14001840 0e000000 62727574  .......@....brut
   1f6c: 6f6d6172 67650000                    omarge..

  Item   6 -- Length:   24  Offset: 8004 (0x1f44)  Flags: USED
   1f44: 00003b0d 3d001840 0e000000 62727574  ..;.=..@....brut
   1f54: 6f6d6172 67650000                    omarge..

  Item   7 -- Length:   24  Offset: 7980 (0x1f2c)  Flags: USED
   1f2c: 00007618 19001840 0e000000 62727574  ..v....@....brut
   1f3c: 6f6d6172 67650000                    omarge..

  Item   8 -- Length:   24  Offset: 7956 (0x1f14)  Flags: USED
   1f14: 0000af19 25001840 0e000000 62727574  ....%..@....brut
   1f24: 6f6d6172 67650000                    omarge..

  Item   9 -- Length:   24  Offset: 7932 (0x1efc)  Flags: USED
   1efc: 00007c1e 0e001840 0e000000 62727574  ..|....@....brut
   1f0c: 6f6d6172 67650000                    omarge..

  Item  10 -- Length:   24  Offset: 7908 (0x1ee4)  Flags: USED
   1ee4: 00001420 11001840 0e000000 62727574  ... ...@....brut
   1ef4: 6f6d6172 67650000                    omarge..

  Item  11 -- Length:   24  Offset: 7884 (0x1ecc)  Flags: USED
   1ecc: 00007528 2a001840 0e000000 62727574  ..u(*..@....brut
   1edc: 6f6d6172 67650000                    omarge..

  Item  12 -- Length:   24  Offset: 7860 (0x1eb4)  Flags: USED
   1eb4: 0000c231 1f001840 0e000000 62727574  ...1...@....brut
   1ec4: 6f6d6172 67650000                    omarge..

  Item  13 -- Length:   24  Offset: 7836 (0x1e9c)  Flags: USED
   1e9c: 00008240 21001840 0e000000 62727574  ...@!..@....brut
   1eac: 6f6d6172 67650000                    omarge..

  Item  14 -- Length:   24  Offset: 7812 (0x1e84)  Flags: USED
   1e84: 0000f45e 39001840 0e000000 62727574  ...^9..@....brut
   1e94: 6f6d6172 67650000                    omarge..

  Item  15 -- Length:   24  Offset: 7788 (0x1e6c)  Flags: USED
   1e6c: 00009362 24001840 0e000000 62727574  ...b$..@....brut
   1e7c: 6f6d6172 67650000                    omarge..

  Item  16 -- Length:   24  Offset: 7764 (0x1e54)  Flags: USED
   1e54: 0000b666 3d001840 0e000000 62727574  ...f=..@....brut
   1e64: 6f6d6172 67650000                    omarge..

  Item  17 -- Length:   24  Offset: 7740 (0x1e3c)  Flags: USED
   1e3c: 0000b76b 1b001840 0e000000 62727574  ...k...@....brut
   1e4c: 6f6d6172 67650000                    omarge..

  Item  18 -- Length:   24  Offset: 7716 (0x1e24)  Flags: USED
   1e24: 00005d71 07001840 0e000000 62727574  ..]q...@....brut
   1e34: 6f6d6172 67650000                    omarge..

  Item  19 -- Length:   24  Offset: 7692 (0x1e0c)  Flags: USED
   1e0c: 00009571 0b001840 0e000000 62727574  ...q...@....brut
   1e1c: 6f6d6172 67650000                    omarge..

  Item  20 -- Length:   24  Offset: 7668 (0x1df4)  Flags: USED
   1df4: 0000ca89 20001840 0e000000 62727574  .... ..@....brut
   1e04: 6f6d6172 67650000                    omarge..

  Item  21 -- Length:   24  Offset: 7644 (0x1ddc)  Flags: USED
   1ddc: 0000608b 0d001840 0e000000 62727574  ..`....@....brut
   1dec: 6f6d6172 67650000                    omarge..

  Item  22 -- Length:   24  Offset: 7620 (0x1dc4)  Flags: USED
   1dc4: 0000f78c 1f001840 0e000000 62727574  .......@....brut
   1dd4: 6f6d6172 67650000                    omarge..

  Item  23 -- Length:   24  Offset: 7596 (0x1dac)  Flags: USED
   1dac: 000065b0 16001840 0e000000 62727574  ..e....@....brut
   1dbc: 6f6d6172 67650000                    omarge..

  Item  24 -- Length:   24  Offset: 7572 (0x1d94)  Flags: USED
   1d94: 00004ab1 22001840 0e000000 62727574  ..J."..@....brut
   1da4: 6f6d6172 67650000                    omarge..

  Item  25 -- Length:   24  Offset: 7548 (0x1d7c)  Flags: USED
   1d7c: 00008d62 0f001840 0f000000 62727574  ...b...@....brut
   1d8c: 6f6d6172 67653b00                    omarge;.

  Item  26 -- Length:   24  Offset: 7524 (0x1d64)  Flags: USED
   1d64: 00001620 22001840 0f000000 62727574  ... "..@....brut
   1d74: 6f6d6172 67657300                    omarges.

  Item  27 -- Length:   24  Offset: 7500 (0x1d4c)  Flags: USED
   1d4c: 00001b69 1f001840 10000000 62727574  ...i...@....brut
   1d5c: 6f6d696e 696d756d                    ominimum

  Item  28 -- Length:   36  Offset: 7464 (0x1d28)  Flags: USED
   1d28: 00008d35 2c002440 1a000000 62727574  ...5,.$@....brut
   1d38: 6f6d696e 696d756d 6a657567 646c6f6e  ominimumjeugdlon
   1d48: 656e0000                             en..

  Item  29 -- Length:   36  Offset: 7428 (0x1d04)  Flags: USED
   1d04: 00001e75 3b002440 1a000000 62727574  ...u;.$@....brut
   1d14: 6f6d696e 696d756d 6a657567 646c6f6e  ominimumjeugdlon
   1d24: 656e0000                             en..

  Item  30 -- Length:   32  Offset: 7396 (0x1ce4)  Flags: USED
   1ce4: 00009974 39002040 16000000 62727574  ...t9. @....brut
   1cf4: 6f2d6d69 6e696d75 6d6c6f6e 656e0000  o-minimumlonen..

  Item  31 -- Length:   28  Offset: 7368 (0x1cc8)  Flags: USED
   1cc8: 0000181f 3f001c40 14000000 62727574  ....?..@....brut
   1cd8: 6f6d696e 696d756d 6c6f6f6e           ominimumloon

  Item  32 -- Length:   32  Offset: 7336 (0x1ca8)  Flags: USED
   1ca8: 0000b220 1c002040 15000000 62727574  ... .. @....brut
   1cb8: 6f2d6d69 6e696d75 6d6c6f6f 6e000000  o-minimumloon...

  Item  33 -- Length:   24  Offset: 7312 (0x1c90)  Flags: USED
   1c90: 00002283 32001840 0f000000 62727574  ..".2..@....brut
   1ca0: 6f2d6e65 74746f00                    o-netto.

  Item  34 -- Length:   36  Offset: 7276 (0x1c6c)  Flags: USED
   1c6c: 0000e39f 01002440 1b000000 62727574  ......$@....brut
   1c7c: 6f2d6e65 74746f72 656b656e 6d616368  o-nettorekenmach
   1c8c: 696e6500                             ine.

  Item  35 -- Length:   24  Offset: 7252 (0x1c54)  Flags: USED
   1c54: 0000a81c 3c001840 0f000000 62727574  ....<..@....brut
   1c64: 6f2d6f6d 7a657400                    o-omzet.

  Item  36 -- Length:   28  Offset: 4844 (0x12ec)  Flags: USED
   12ec: 00001ecb 1a001c40 13000000 62727574  .......@....brut
   12fc: 6f2d6f70 6272656e 67737400           o-opbrengst.

  Item  37 -- Length:   28  Offset: 7224 (0x1c38)  Flags: USED
   1c38: 00004e1d 22001c40 13000000 62727574  ..N."..@....brut
   1c48: 6f2d6f70 6272656e 67737400           o-opbrengst.

  Item  38 -- Length:   28  Offset: 7196 (0x1c1c)  Flags: USED
   1c1c: 000025a3 22001c40 13000000 62727574  ..%."..@....brut
   1c2c: 6f2d6f70 6272656e 67737400           o-opbrengst.

  Item  39 -- Length:   32  Offset: 7164 (0x1bfc)  Flags: USED
   1bfc: 00003008 32002040 15000000 62727574  ..0.2. @....brut
   1c0c: 6f2d6f70 6272656e 67737465 6e000000  o-opbrengsten...

  Item  40 -- Length:   32  Offset: 7132 (0x1bdc)  Flags: USED
   1bdc: 0000ac1f 31002040 15000000 62727574  ....1. @....brut
   1bec: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  41 -- Length:   32  Offset: 7100 (0x1bbc)  Flags: USED
   1bbc: 0000f225 2a002040 15000000 62727574  ...%*. @....brut
   1bcc: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  42 -- Length:   32  Offset: 7068 (0x1b9c)  Flags: USED
   1b9c: 0000284f 3c002040 15000000 62727574  ..(O<. @....brut
   1bac: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  43 -- Length:   32  Offset: 7036 (0x1b7c)  Flags: USED
   1b7c: 00002f55 19002040 15000000 62727574  ../U.. @....brut
   1b8c: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  44 -- Length:   32  Offset: 7004 (0x1b5c)  Flags: USED
   1b5c: 00003155 17002040 15000000 62727574  ..1U.. @....brut
   1b6c: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  45 -- Length:   32  Offset: 6972 (0x1b3c)  Flags: USED
   1b3c: 00006b75 09002040 15000000 62727574  ..ku.. @....brut
   1b4c: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  46 -- Length:   32  Offset: 6940 (0x1b1c)  Flags: USED
   1b1c: 0000ac88 18002040 15000000 62727574  ...... @....brut
   1b2c: 6f2d6f70 70657276 6c616b74 65000000  o-oppervlakte...

  Item  47 -- Length:   28  Offset: 6912 (0x1b00)  Flags: USED
   1b00: 000038ba 02001c40 11000000 62727574  ..8....@....brut
   1b10: 6f70656e 73696f65 6e000000           opensioen...

  Item  48 -- Length:   24  Offset: 6888 (0x1ae8)  Flags: USED
   1ae8: 0000db0d 27001840 0f000000 62727574  ....'..@....brut
   1af8: 6f707265 6d696500                    opremie.

  Item  49 -- Length:   24  Offset: 6864 (0x1ad0)  Flags: USED
   1ad0: 00002a18 10001840 0f000000 62727574  ..*....@....brut
   1ae0: 6f707265 6d696500                    opremie.

  Item  50 -- Length:   24  Offset: 6840 (0x1ab8)  Flags: USED
   1ab8: 00006541 12001840 0f000000 62727574  ..eA...@....brut
   1ac8: 6f707265 6d696500                    opremie.

  Item  51 -- Length:   24  Offset: 6816 (0x1aa0)  Flags: USED
   1aa0: 00007262 12001840 0e000000 62727574  ..rb...@....brut
   1ab0: 6f707269 6a730000                    oprijs..

  Item  52 -- Length:   28  Offset: 6788 (0x1a84)  Flags: USED
   1a84: 0000c413 24001c40 12000000 62727574  ....$..@....brut
   1a94: 6f72656e 64656d65 6e740000           orendement..

  Item  53 -- Length:   28  Offset: 6760 (0x1a68)  Flags: USED
   1a68: 0000ea09 3c001c40 13000000 62727574  ....<..@....brut
   1a78: 6f2d7265 6e64656d 656e7400           o-rendement.

  Item  54 -- Length:   24  Offset: 6736 (0x1a50)  Flags: USED
   1a50: 0000d54c 28001840 0e000000 62727574  ...L(..@....brut
   1a60: 6f72656e 74650000                    orente..

  Item  55 -- Length:   36  Offset: 6700 (0x1a2c)  Flags: USED
   1a2c: 000086c1 2b002440 19000000 62727574  ....+.$@....brut
   1a3c: 6f2d7265 6e74652d 696e6b6f 6d737465  o-rente-inkomste
   1a4c: 6e000000                             n...

  Item  56 -- Length:   32  Offset: 6668 (0x1a0c)  Flags: USED
   1a0c: 000086c1 23002040 18000000 62727574  ....#. @....brut
   1a1c: 6f72656e 74652d69 6e6b6f6d 7374656e  orente-inkomsten

  Item  57 -- Length:   24  Offset: 4796 (0x12bc)  Flags: USED
   12bc: 00001fcc 13001840 10000000 62727574  .......@....brut
   12cc: 6f73616c 61726973                    osalaris

  Item  58 -- Length:   24  Offset: 4820 (0x12d4)  Flags: USED
   12d4: 0000fbca 2b001840 10000000 62727574  ....+..@....brut
   12e4: 6f73616c 61726973                    osalaris

  Item  59 -- Length:   24  Offset: 4872 (0x1308)  Flags: USED
   1308: 000008ca 1b001840 10000000 62727574  .......@....brut
   1318: 6f73616c 61726973                    osalaris

  Item  60 -- Length:   24  Offset: 6644 (0x19f4)  Flags: USED
   19f4: 000035c3 27001840 10000000 62727574  ..5.'..@....brut
   1a04: 6f73616c 61726973                    osalaris

  Item  61 -- Length:   24  Offset: 6620 (0x19dc)  Flags: USED
   19dc: 000069c7 25001840 10000000 62727574  ..i.%..@....brut
   19ec: 6f73616c 61726973                    osalaris

  Item  62 -- Length:   24  Offset: 6596 (0x19c4)  Flags: USED
   19c4: 000042c5 31001840 10000000 62727574  ..B.1..@....brut
   19d4: 6f73616c 61726973                    osalaris

  Item  63 -- Length:   24  Offset: 6572 (0x19ac)  Flags: USED
   19ac: 0000cbbe 24001840 10000000 62727574  ....$..@....brut
   19bc: 6f73616c 61726973                    osalaris

  Item  64 -- Length:   24  Offset: 6548 (0x1994)  Flags: USED
   1994: 0000dec2 0e001840 10000000 62727574  .......@....brut
   19a4: 6f73616c 61726973                    osalaris

  Item  65 -- Length:   24  Offset: 6524 (0x197c)  Flags: USED
   197c: 0000d8c0 18001840 10000000 62727574  .......@....brut
   198c: 6f73616c 61726973                    osalaris

  Item  66 -- Length:   24  Offset: 6500 (0x1964)  Flags: USED
   1964: 00000dc2 05001840 10000000 62727574  .......@....brut
   1974: 6f73616c 61726973                    osalaris

  Item  67 -- Length:   24  Offset: 6476 (0x194c)  Flags: USED
   194c: 00004380 1e001840 10000000 62727574  ..C....@....brut
   195c: 6f73616c 61726973                    osalaris

  Item  68 -- Length:   24  Offset: 6452 (0x1934)  Flags: USED
   1934: 000027bb 17001840 10000000 62727574  ..'....@....brut
   1944: 6f73616c 61726973                    osalaris

  Item  69 -- Length:   24  Offset: 6428 (0x191c)  Flags: USED
   191c: 00000806 0b001840 10000000 62727574  .......@....brut
   192c: 6f73616c 61726973                    osalaris

  Item  70 -- Length:   24  Offset: 6404 (0x1904)  Flags: USED
   1904: 00009207 0f001840 10000000 62727574  .......@....brut
   1914: 6f73616c 61726973                    osalaris

  Item  71 -- Length:   24  Offset: 6380 (0x18ec)  Flags: USED
   18ec: 00004809 29001840 10000000 62727574  ..H.)..@....brut
   18fc: 6f73616c 61726973                    osalaris

  Item  72 -- Length:   24  Offset: 6356 (0x18d4)  Flags: USED
   18d4: 0000bb09 23001840 10000000 62727574  ....#..@....brut
   18e4: 6f73616c 61726973                    osalaris

  Item  73 -- Length:   24  Offset: 6332 (0x18bc)  Flags: USED
   18bc: 0000710a 31001840 10000000 62727574  ..q.1..@....brut
   18cc: 6f73616c 61726973                    osalaris

  Item  74 -- Length:   24  Offset: 6308 (0x18a4)  Flags: USED
   18a4: 0000e00d 35001840 10000000 62727574  ....5..@....brut
   18b4: 6f73616c 61726973                    osalaris

  Item  75 -- Length:   24  Offset: 6284 (0x188c)  Flags: USED
   188c: 00006611 01001840 10000000 62727574  ..f....@....brut
   189c: 6f73616c 61726973                    osalaris

  Item  76 -- Length:   24  Offset: 6260 (0x1874)  Flags: USED
   1874: 0000aa12 0d001840 10000000 62727574  .......@....brut
   1884: 6f73616c 61726973                    osalaris

  Item  77 -- Length:   24  Offset: 6236 (0x185c)  Flags: USED
   185c: 00002d13 3d001840 10000000 62727574  ..-.=..@....brut
   186c: 6f73616c 61726973                    osalaris

  Item  78 -- Length:   24  Offset: 6212 (0x1844)  Flags: USED
   1844: 00001316 39001840 10000000 62727574  ....9..@....brut
   1854: 6f73616c 61726973                    osalaris

  Item  79 -- Length:   24  Offset: 6188 (0x182c)  Flags: USED
   182c: 00006018 08001840 10000000 62727574  ..`....@....brut
   183c: 6f73616c 61726973                    osalaris

  Item  80 -- Length:   24  Offset: 6164 (0x1814)  Flags: USED
   1814: 0000811a 0c001840 10000000 62727574  .......@....brut
   1824: 6f73616c 61726973                    osalaris

  Item  81 -- Length:   24  Offset: 6140 (0x17fc)  Flags: USED
   17fc: 0000ba1e 2f001840 10000000 62727574  ..../..@....brut
   180c: 6f73616c 61726973                    osalaris

  Item  82 -- Length:   24  Offset: 6116 (0x17e4)  Flags: USED
   17e4: 00007120 1b001840 10000000 62727574  ..q ...@....brut
   17f4: 6f73616c 61726973                    osalaris

  Item  83 -- Length:   24  Offset: 6092 (0x17cc)  Flags: USED
   17cc: 0000aa23 0b001840 10000000 62727574  ...#...@....brut
   17dc: 6f73616c 61726973                    osalaris

  Item  84 -- Length:   24  Offset: 6068 (0x17b4)  Flags: USED
   17b4: 00002930 09001840 10000000 62727574  ..)0...@....brut
   17c4: 6f73616c 61726973                    osalaris

  Item  85 -- Length:   24  Offset: 6044 (0x179c)  Flags: USED
   179c: 0000b531 32001840 10000000 62727574  ...12..@....brut
   17ac: 6f73616c 61726973                    osalaris

  Item  86 -- Length:   24  Offset: 6020 (0x1784)  Flags: USED
   1784: 00000c32 05001840 10000000 62727574  ...2...@....brut
   1794: 6f73616c 61726973                    osalaris

  Item  87 -- Length:   24  Offset: 5996 (0x176c)  Flags: USED
   176c: 00008732 29001840 10000000 62727574  ...2)..@....brut
   177c: 6f73616c 61726973                    osalaris

  Item  88 -- Length:   24  Offset: 5972 (0x1754)  Flags: USED
   1754: 00009e32 15001840 10000000 62727574  ...2...@....brut
   1764: 6f73616c 61726973                    osalaris

  Item  89 -- Length:   24  Offset: 5948 (0x173c)  Flags: USED
   173c: 0000cb32 2d001840 10000000 62727574  ...2-..@....brut
   174c: 6f73616c 61726973                    osalaris

  Item  90 -- Length:   24  Offset: 5924 (0x1724)  Flags: USED
   1724: 0000e833 05001840 10000000 62727574  ...3...@....brut
   1734: 6f73616c 61726973                    osalaris

  Item  91 -- Length:   24  Offset: 5900 (0x170c)  Flags: USED
   170c: 0000d034 20001840 10000000 62727574  ...4 ..@....brut
   171c: 6f73616c 61726973                    osalaris

  Item  92 -- Length:   24  Offset: 5876 (0x16f4)  Flags: USED
   16f4: 0000a136 14001840 10000000 62727574  ...6...@....brut
   1704: 6f73616c 61726973                    osalaris

  Item  93 -- Length:   24  Offset: 5852 (0x16dc)  Flags: USED
   16dc: 0000eb38 14001840 10000000 62727574  ...8...@....brut
   16ec: 6f73616c 61726973                    osalaris

  Item  94 -- Length:   24  Offset: 5828 (0x16c4)  Flags: USED
   16c4: 00006c41 33001840 10000000 62727574  ..lA3..@....brut
   16d4: 6f73616c 61726973                    osalaris

  Item  95 -- Length:   24  Offset: 5804 (0x16ac)  Flags: USED
   16ac: 00008d43 16001840 10000000 62727574  ...C...@....brut
   16bc: 6f73616c 61726973                    osalaris

  Item  96 -- Length:   24  Offset: 5780 (0x1694)  Flags: USED
   1694: 0000e66a 19001840 10000000 62727574  ...j...@....brut
   16a4: 6f73616c 61726973                    osalaris

  Item  97 -- Length:   24  Offset: 5756 (0x167c)  Flags: USED
   167c: 0000786c 06001840 10000000 62727574  ..xl...@....brut
   168c: 6f73616c 61726973                    osalaris

  Item  98 -- Length:   24  Offset: 5732 (0x1664)  Flags: USED
   1664: 0000426d 18001840 10000000 62727574  ..Bm...@....brut
   1674: 6f73616c 61726973                    osalaris

  Item  99 -- Length:   24  Offset: 5708 (0x164c)  Flags: USED
   164c: 00008b6e 1e001840 10000000 62727574  ...n...@....brut
   165c: 6f73616c 61726973                    osalaris

  Item 100 -- Length:   24  Offset: 5684 (0x1634)  Flags: USED
   1634: 0000976e 11001840 10000000 62727574  ...n...@....brut
   1644: 6f73616c 61726973                    osalaris

  Item 101 -- Length:   24  Offset: 5660 (0x161c)  Flags: USED
   161c: 0000086f 28001840 10000000 62727574  ...o(..@....brut
   162c: 6f73616c 61726973                    osalaris

  Item 102 -- Length:   24  Offset: 5636 (0x1604)  Flags: USED
   1604: 0000c86f 22001840 10000000 62727574  ...o"..@....brut
   1614: 6f73616c 61726973                    osalaris

  Item 103 -- Length:   24  Offset: 5612 (0x15ec)  Flags: USED
   15ec: 00004770 11001840 10000000 62727574  ..Gp...@....brut
   15fc: 6f73616c 61726973                    osalaris

  Item 104 -- Length:   24  Offset: 5588 (0x15d4)  Flags: USED
   15d4: 00005270 3b001840 10000000 62727574  ..Rp;..@....brut
   15e4: 6f73616c 61726973                    osalaris

  Item 105 -- Length:   24  Offset: 5564 (0x15bc)  Flags: USED
   15bc: 00007373 3e001840 10000000 62727574  ..ss>..@....brut
   15cc: 6f73616c 61726973                    osalaris

  Item 106 -- Length:   24  Offset: 5540 (0x15a4)  Flags: USED
   15a4: 00008b7f 17001840 10000000 62727574  .......@....brut
   15b4: 6f73616c 61726973                    osalaris

  Item 107 -- Length:   24  Offset: 5516 (0x158c)  Flags: USED
   158c: 00002283 01001840 10000000 62727574  .."....@....brut
   159c: 6f73616c 61726973                    osalaris

  Item 108 -- Length:   24  Offset: 5492 (0x1574)  Flags: USED
   1574: 00002988 1f001840 10000000 62727574  ..)....@....brut
   1584: 6f73616c 61726973                    osalaris

  Item 109 -- Length:   24  Offset: 5468 (0x155c)  Flags: USED
   155c: 0000a68b 13001840 10000000 62727574  .......@....brut
   156c: 6f73616c 61726973                    osalaris

  Item 110 -- Length:   24  Offset: 5444 (0x1544)  Flags: USED
   1544: 0000c492 2b001840 10000000 62727574  ....+..@....brut
   1554: 6f73616c 61726973                    osalaris

  Item 111 -- Length:   24  Offset: 5420 (0x152c)  Flags: USED
   152c: 00004a9f 20001840 10000000 62727574  ..J. ..@....brut
   153c: 6f73616c 61726973                    osalaris

  Item 112 -- Length:   24  Offset: 5396 (0x1514)  Flags: USED
   1514: 000035a4 30001840 10000000 62727574  ..5.0..@....brut
   1524: 6f73616c 61726973                    osalaris

  Item 113 -- Length:   24  Offset: 5372 (0x14fc)  Flags: USED
   14fc: 000041a4 24001840 10000000 62727574  ..A.$..@....brut
   150c: 6f73616c 61726973                    osalaris

  Item 114 -- Length:   24  Offset: 5348 (0x14e4)  Flags: USED
   14e4: 000010ab 03001840 10000000 62727574  .......@....brut
   14f4: 6f73616c 61726973                    osalaris

  Item 115 -- Length:   24  Offset: 5324 (0x14cc)  Flags: USED
   14cc: 00004db2 3e001840 10000000 62727574  ..M.>..@....brut
   14dc: 6f73616c 61726973                    osalaris

  Item 116 -- Length:   24  Offset: 5300 (0x14b4)  Flags: USED
   14b4: 0000cab5 05001840 10000000 62727574  .......@....brut
   14c4: 6f73616c 61726973                    osalaris

  Item 117 -- Length:   24  Offset: 5276 (0x149c)  Flags: USED
   149c: 000005b6 2e001840 10000000 62727574  .......@....brut
   14ac: 6f73616c 61726973                    osalaris

  Item 118 -- Length:   28  Offset: 5248 (0x1480)  Flags: USED
   1480: 00006611 09001c40 11000000 62727574  ..f....@....brut
   1490: 6f2d7361 6c617269 73000000           o-salaris...

  Item 119 -- Length:   28  Offset: 5220 (0x1464)  Flags: USED
   1464: 0000901b 0b001c40 11000000 62727574  .......@....brut
   1474: 6f2d7361 6c617269 73000000           o-salaris...

  Item 120 -- Length:   28  Offset: 5192 (0x1448)  Flags: USED
   1448: 0000362e 1e001c40 11000000 62727574  ..6....@....brut
   1458: 6f2d7361 6c617269 73000000           o-salaris...

  Item 121 -- Length:   28  Offset: 5164 (0x142c)  Flags: USED
   142c: 0000532f 0a001c40 11000000 62727574  ..S/...@....brut
   143c: 6f2d7361 6c617269 73000000           o-salaris...

  Item 122 -- Length:   28  Offset: 5136 (0x1410)  Flags: USED
   1410: 0000be70 35001c40 11000000 62727574  ...p5..@....brut
   1420: 6f2d7361 6c617269 73000000           o-salaris...

  Item 123 -- Length:   28  Offset: 5108 (0x13f4)  Flags: USED
   13f4: 00002593 3e001c40 11000000 62727574  ..%.>..@....brut
   1404: 6f2d7361 6c617269 73000000           o-salaris...

  Item 124 -- Length:   40  Offset: 5068 (0x13cc)  Flags: USED
   13cc: 0000c14a 2c002840 1d000000 62727574  ...J,.(@....brut
   13dc: 6f2d7361 6c617269 73766572 67656c69  o-salarisvergeli
   13ec: 6a6b696e 67000000                    jking...

  Item 125 -- Length:   36  Offset: 5032 (0x13a8)  Flags: USED
   13a8: 00000f45 05002440 1a000000 62727574  ...E..$@....brut
   13b8: 6f2d7363 68616465 76657267 6f656469  o-schadevergoedi
   13c8: 6e670000                             ng..

  Item 126 -- Length:   40  Offset: 4992 (0x1380)  Flags: USED
   1380: 0000d029 12002840 1e000000 62727574  ...)..(@....brut
   1390: 6f737469 6b73746f 6670726f 64756374  ostikstofproduct
   13a0: 69656e6f 726d0000                    ienorm..

  Item 127 -- Length:   40  Offset: 4952 (0x1358)  Flags: USED
   1358: 0000d029 32002840 20000000 62727574  ...)2.(@ ...brut
   1368: 6f737469 6b73746f 6670726f 64756374  ostikstofproduct
   1378: 69656e6f 726d656e                    ienormen

  Item 128 -- Length:   28  Offset: 4924 (0x133c)  Flags: USED
   133c: 000039ba 1b001c40 12000000 62727574  ..9....@....brut
   134c: 6f756974 6b657269 6e670000           ouitkering..

  Item 129 -- Length:   28  Offset: 4896 (0x1320)  Flags: USED
   1320: 0000382e 22001c40 13000000 62727574  ..8."..@....brut
   1330: 6f2d7569 746b6572 696e6700           o-uitkering.


<Special Section> -----
  BTree Index Section:
   Flags: 0x0001 (LEAF)
   Blocks: Previous (12811)  Next (12769)  Level (0)

   1ff0: 0b320000 e1310000 00000000 01000000  .2...1..........


*** End of Requested Range Encountered. Last Block Read: 39248 ***

Tom Lane wrote:
> Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
>
>>I asked the developper to delete all imported data again an restart the
>>import.  This import crashed again with the same error but this time on
>>another block.
>
>
>>2006-03-27 00:15:25.458 CESTERROR:  XX001: invalid page header in block
>>48068 of relation "dunn_main"
>>2006-03-27 00:15:25.458 CESTCONTEXT:  SQL statement "SELECT  phone FROM
>>dunn_main WHERE source_id =  $1  AND duns =  $2 "
>>         PL/pgSQL function "proc_dunn" line 29 at select into variables
>>2006-03-27 00:15:25.458 CESTLOCATION:  ReadBuffer, bufmgr.c:257
>>2006-03-27 00:15:25.458 CESTSTATEMENT:  SELECT proc_dunn ('J M
>>Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
>>54,387166)
>
>
>>But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
>>3BU','215517942','1','01','S',NULL,'0219',156,1
>>54,387166)' statement now, it works without errors.
>
>
> That is *really* strange.  Are you certain that the function is
> examining the same table you are?  I'm wondering about multiple
> similarly-named tables in different schemas, or something like that.
>
>
>>If I would like to dump block 48068 now with pg_dumpfile, how do i know
>>which file this block belongs to?
>
>
> See
> http://www.postgresql.org/docs/8.1/static/storage.html
> and/or use contrib/oid2name.
>
>             regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match
>

OK.  The saga continues, everything is a little bit more clear, but at
the same time a lot more confusing.

Today i wanted to reproduce the problem again.  And guess what? A vacuum
of the database went thru without any problems.

I dump the block i was having problems with yesterday.  It doesn't
report an invalid header anymore and it contains other data!!!

Turns out the data that was returned yesterday belongs to another database!

Some more detail about the setup.  This server runs 2 instances of
postgresql.  One production instance which is version 8.0.3.  And
another testing instance installed in a different folder which runs
version 8.1.3  Am I wrong thinking this setup ought to work?

Both instances use completely seperated data folders.

So the first dump returned data that actually belongs to an 8.0.3
database (that runs fine).  And today without _any_ intervention that
same block returns the correct data and the complete database is fine.

Where is the problem?
    The fact that i'm running 2 different instances?
    Cache on raid controller messing up?
    Some strange voodoo?




Jo De Haes wrote:
> Ok,  So we reran everything and got the same error message again, now
> i'm able to reproduce it.

>
> Tom Lane wrote:
>
>> Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
>>
>>> I asked the developper to delete all imported data again an restart
>>> the import.  This import crashed again with the same error but this
>>> time on another block.
>>
>>
>>
>>> 2006-03-27 00:15:25.458 CESTERROR:  XX001: invalid page header in
>>> block 48068 of relation "dunn_main"
>>> 2006-03-27 00:15:25.458 CESTCONTEXT:  SQL statement "SELECT  phone
>>> FROM dunn_main WHERE source_id =  $1  AND duns =  $2 "
>>>         PL/pgSQL function "proc_dunn" line 29 at select into variables
>>> 2006-03-27 00:15:25.458 CESTLOCATION:  ReadBuffer, bufmgr.c:257
>>> 2006-03-27 00:15:25.458 CESTSTATEMENT:  SELECT proc_dunn ('J M
>>> Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
>>> 54,387166)
>>
>>
>>
>>> But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
>>> 3BU','215517942','1','01','S',NULL,'0219',156,1
>>> 54,387166)' statement now, it works without errors.
>>
>>
>>
>> That is *really* strange.  Are you certain that the function is
>> examining the same table you are?  I'm wondering about multiple
>> similarly-named tables in different schemas, or something like that.
>>
>>
>>> If I would like to dump block 48068 now with pg_dumpfile, how do i
>>> know which file this block belongs to?
>>
>>
>>
>> See
>> http://www.postgresql.org/docs/8.1/static/storage.html
>> and/or use contrib/oid2name.
>>
>>             regards, tom lane
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>>        choose an index scan if your joining column's datatypes do not
>>        match
>>

Jo De Haes wrote:

> OK.  The saga continues, everything is a little bit more clear, but at
> the same time a lot more confusing.
>
> Today i wanted to reproduce the problem again.  And guess what? A
> vacuum of the database went thru without any problems.
>
> I dump the block i was having problems with yesterday.  It doesn't
> report an invalid header anymore and it contains other data!!!
>
Inconsistant problems esp. with PostgreSQL are usually the result of
hardware failure.

> Turns out the data that was returned yesterday belongs to another
> database!
>
> Some more detail about the setup.  This server runs 2 instances of
> postgresql.  One production instance which is version 8.0.3.  And
> another testing instance installed in a different folder which runs
> version 8.1.3  Am I wrong thinking this setup ought to work?

No.  Ihave done it before too.  PostgreSQL instances running on
different ports or addresses are sufficiently isolated to prevent this
from being a problem.

>
> Both instances use completely seperated data folders.
>
> So the first dump returned data that actually belongs to an 8.0.3
> database (that runs fine).  And today without _any_ intervention that
> same block returns the correct data and the complete database is fine.
>
> Where is the problem?
>     The fact that i'm running 2 different instances?
>     Cache on raid controller messing up?
>     Some strange voodoo?

I would see what sort of memory testing suite you can run on your system
first (memtestx86, for example) and go from there.  It sounds to me like
some sort of a hardware issue.  It *could* be bits flipped anywhere,
from the writehead on the disk to the main system memory or the CPU.

The likelihood that it is a random RAM error is reduced if you are using
ECC RAM.  Otherwise it could be anything.

This being said, when I have seen bits flipped by the CPU usually you
get a lot of index issues and shared memory corruptions, so I would be
more inclined to think that this was RAM or RAID cache.

Best Wishes,
Chris Travers
Metatron Technology Consulting

Just a little followup on this problem.

We've moved the database to another server where it ran without problems.

HP just released new raid controller drivers for Suse and a firmware
update for the controller itself.

Until now the problem hasn't occurred anymore.

Thanks!
Jo.

Chris Travers wrote:
> Jo De Haes wrote:
>
>> OK.  The saga continues, everything is a little bit more clear, but at
>> the same time a lot more confusing.
>>
>> Today i wanted to reproduce the problem again.  And guess what? A
>> vacuum of the database went thru without any problems.
>>
>> I dump the block i was having problems with yesterday.  It doesn't
>> report an invalid header anymore and it contains other data!!!
>>
> Inconsistant problems esp. with PostgreSQL are usually the result of
> hardware failure.
>
>> Turns out the data that was returned yesterday belongs to another
>> database!
>>
>> Some more detail about the setup.  This server runs 2 instances of
>> postgresql.  One production instance which is version 8.0.3.  And
>> another testing instance installed in a different folder which runs
>> version 8.1.3  Am I wrong thinking this setup ought to work?
>
>
> No.  Ihave done it before too.  PostgreSQL instances running on
> different ports or addresses are sufficiently isolated to prevent this
> from being a problem.
>
>>
>> Both instances use completely seperated data folders.
>>
>> So the first dump returned data that actually belongs to an 8.0.3
>> database (that runs fine).  And today without _any_ intervention that
>> same block returns the correct data and the complete database is fine.
>>
>> Where is the problem?
>>     The fact that i'm running 2 different instances?
>>     Cache on raid controller messing up?
>>     Some strange voodoo?
>
>
> I would see what sort of memory testing suite you can run on your system
> first (memtestx86, for example) and go from there.  It sounds to me like
> some sort of a hardware issue.  It *could* be bits flipped anywhere,
> from the writehead on the disk to the main system memory or the CPU.
>
> The likelihood that it is a random RAM error is reduced if you are using
> ECC RAM.  Otherwise it could be anything.
>
> This being said, when I have seen bits flipped by the CPU usually you
> get a lot of index issues and shared memory corruptions, so I would be
> more inclined to think that this was RAM or RAID cache.
>
> Best Wishes,
> Chris Travers
> Metatron Technology Consulting
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq