Hey everyone,
We have a PG 8.3.7 server that is doing WAL log shipping to 2 other servers that are remote mirrors. This has been working well for almost two years. Last night we did some massive data and structure changes to one of our databases. Since then I get these errors on the two mirrors:
2010-09-15 08:35:05 EDT: LOG: restored log file "0000000100000301000000D9" from archive
2010-09-15 08:35:27 EDT: LOG: restored log file "0000000100000301000000DA" from archive
2010-09-15 08:35:40 EDT: LOG: restored log file "0000000100000301000000DB" from archive
2010-09-15 08:35:40 EDT: LOG: unexpected pageaddr 301/47000000 in log file 769, segment 219, offset 0
2010-09-15 08:35:40 EDT: LOG: redo done at 301/DA370780
2010-09-15 08:35:40 EDT: LOG: last completed transaction was at log time 2010-09-15 08:30:01.24936-04
2010-09-15 08:35:40 EDT: LOG: restored log file "0000000100000301000000DA" from archive
2010-09-15 08:36:26 EDT: LOG: selected new timeline ID: 2
2010-09-15 08:37:11 EDT: LOG: archive recovery complete
I've taken two separate file level backups and tried to restart the mirrors, and every time on both servers I get a similar error message. I seem to recall reading that it may have something to do with corruption in the timeline, which is why it's jumping to a new timeline ID.
1. Can anyone tell me what this means?
2. Is there some corruption in the database?
3. If so, is there an easy way to fix it?
Also, one additional question. I don't have a 00001.history file which makes the PITRTools complain constantly. Is there any way to regenerate this file?
Any help would be much appreciated. I'm rather worried that I've got corruption, and not having the mirrors running puts us at risk for data loss.