On 2014-06-02 18:17:46 +0100, Greg Stark wrote:
> Ok, not sure why my first attempt didn't turn this up. I found the
> split in segment 334/91:
>
> rmgr: Btree len (rec/tot): 3776/ 9220, tx: 95765459, lsn:
> 334/91455AB8, prev 334/91455A70, bkp: 0100, desc: split_l: rel
> 1663/16385/1665279 left 175193, right 193740, next 182402, level 0,
> firstright 138
>
> I've attached all the xlog records pertaining to this relation
> grepping for either of these two blocks.
>
> Now interestingly the hot backup was taken starting at 334/90 and
> replay didn't finish until 339/65 so it is entirely possible, even
> likely, that the backup caught this split in an inconsistent state.
Did you check whether all the necessary FPIs were generated? That'd be
my very first suspect.
How many checkpoints are inbetween 334/90 and 339/65?
> How should I go about dumping the two blocks? I have the backup prior
> to WAL replay as well as all the WAL for this time period. I can't
> connect to the database so I'm guessing this will look like replay
> until it hits a record for these block, use dd to extract the block,
> rinse lather repeat. Then dump each of those extracted pages using
> pageinspect on byteas. This sounds pretty laborious :(
I guess you could make xlogdump dump the data from the backup blocks...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services