Обсуждение: help with error "unexpected pageaddr"

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:
> We have a PG 8.3.7 server that is doing WAL log shipping to 2 other
> servers that are remote mirrors.  This has been working well for almost
> two years.  Last night we did some massive data and structure changes to
> one of our databases.  Since then I get these errors on the two mirrors:

> 2010-09-15 08:35:05 EDT: LOG:  restored log file
> "0000000100000301000000D9" from archive

> 2010-09-15 08:35:27 EDT: LOG:  restored log file
> "0000000100000301000000DA" from archive

> 2010-09-15 08:35:40 EDT: LOG:  restored log file
> "0000000100000301000000DB" from archive

> 2010-09-15 08:35:40 EDT: LOG:  unexpected pageaddr 301/47000000 in log
> file 769, segment 219, offset 0

This appears to indicate that you archived the wrong contents of log
file 0000000100000301000000DB.  If you don't still have the correct
contents on the master, I think the only way to recover is to take a
fresh base backup so you can make the slaves roll forward from a point
later than this log segment.  There's no reason to suppose that there's
data corruption on the master, just bad data in the WAL archive.

You'd probably be well advised to look closely at your WAL archiving
script to see if it has any race conditions that might be triggered by
very fast generation of WAL.

> Also, one additional question.  I don't have a 00001.history file which
> makes the PITRTools complain constantly.  Is there any way to regenerate
> this file?

Just ignore that, it's cosmetic (the file isn't supposed to exist).

            regards, tom lane

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:
> I tried to take a new base backup about 45 minutes ago.  The master has
> rolled forward a number of WAL files since I last tried, but it still
> fails.

> LOG:  restored log file "0000000100000301000000FE" from archive
> LOG:  restored log file "000000010000030200000000" from archive
> LOG:  restored log file "000000010000030200000001" from archive
> LOG:  restored log file "000000010000030200000002" from archive
> LOG:  restored log file "000000010000030200000003" from archive
> LOG:  unexpected pageaddr 301/50000000 in log file 770, segment 3,
> offset 0

Hmmm ... is it possible that your WAL archive contains log files
numbered higher than where your master is?

            regards, tom lane

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:
> We have a PG 8.3.7 server that is doing WAL log shipping to 2 other
> servers that are remote mirrors.  This has been working well for
almost
> two years.  Last night we did some massive data and structure changes
to
> one of our databases.  Since then I get these errors on the two
mirrors:

> 2010-09-15 08:35:05 EDT: LOG:  restored log file
> "0000000100000301000000D9" from archive

> 2010-09-15 08:35:27 EDT: LOG:  restored log file
> "0000000100000301000000DA" from archive

> 2010-09-15 08:35:40 EDT: LOG:  restored log file
> "0000000100000301000000DB" from archive

> 2010-09-15 08:35:40 EDT: LOG:  unexpected pageaddr 301/47000000 in log
> file 769, segment 219, offset 0

This appears to indicate that you archived the wrong contents of log
file 0000000100000301000000DB.  If you don't still have the correct
contents on the master, I think the only way to recover is to take a
fresh base backup so you can make the slaves roll forward from a point
later than this log segment.  There's no reason to suppose that there's
data corruption on the master, just bad data in the WAL archive.

You'd probably be well advised to look closely at your WAL archiving
script to see if it has any race conditions that might be triggered by
very fast generation of WAL.

> Also, one additional question.  I don't have a 00001.history file
which
> makes the PITRTools complain constantly.  Is there any way to
regenerate
> this file?

Just ignore that, it's cosmetic (the file isn't supposed to exist).

            regards, tom lane


Tom,

I tried to take a new base backup about 45 minutes ago.  The master has
rolled forward a number of WAL files since I last tried, but it still
fails.

LOG:  restored log file "0000000100000301000000FE" from archive
LOG:  restored log file "000000010000030200000000" from archive
LOG:  restored log file "000000010000030200000001" from archive
LOG:  restored log file "000000010000030200000002" from archive
LOG:  restored log file "000000010000030200000003" from archive
LOG:  unexpected pageaddr 301/50000000 in log file 770, segment 3,
offset 0
LOG:  redo done at 302/2BCE828
LOG:  last completed transaction was at log time 2010-09-15
15:07:01.040854-04
LOG:  restored log file "000000010000030200000002" from archive
LOG:  selected new timeline ID: 2

My entire WAL archiving script is 4 cp %p %f commands.  It's so short I
don't even have a script, it's directly in the postgresql.conf archive
command.

Shouldn't have, the only thing we did to the server was restart it and
run our database queries.  Clearing out all the wal files from pg_xlog
along with a new base backup did fix it though.

Thanks for the help Tom!

Scot Kreienkamp
skreien@la-z-boy.com