Willy-Bas Loos <willybas@gmail.com> writes:
> However, our test database cluster seems to be broken.
> When i try to start the cluster it says:
> 2015-11-16 15:06:35 CET db: ip: us: LOG: database system was interrupted
> while in recovery at 2015-11-16 13:05:41 CET
> 2015-11-16 15:06:35 CET db: ip: us: HINT: This probably means that some
> data is corrupted and you will have to use the last backup for recovery.
> 2015-11-16 15:06:35 CET db: ip: us: LOG: database system was not properly
> shut down; automatic recovery in progress
> 2015-11-16 15:06:35 CET db: ip: us: PANIC: unexpected pageaddr A/6E786000
> in log segment 000000010000000A00000084, offset 7888896
> 2015-11-16 15:06:35 CET db: ip: us: LOG: startup process (PID 11634) was
> terminated by signal 6: Aborted
The offsets match up (7888896 = 0x786000) so evidently what you've got
here is a page in a WAL file that is still from a previous cycle of life,
ie this WAL file used to be 000000010000000A0000006E and has been
recycled, but this particular page hasn't been rewritten yet. The fact
that this is a PANIC and not just normal end of recovery implies that the
pg_control checkpoint WAL-replay-start address is pointing here.
In short, therefore, what you've got here is an indication that a
checkpoint update of pg_control happened before the corresponding WAL had
been flushed to disk. Which means that your filesystem stack is not
honoring fsync semantics properly, because PG would certainly have fsync'd
the WAL first (unless you turned off fsync ...)
So I'd say your test database served its purpose as a coal mine canary,
by letting you know that there's something rotten in your filesystem
and/or RAID hardware. I'd suggest doing actual plug-pull tests when
you think you've got that resolved, rather than just taking it on faith
that it works.
> The cluster is on an OpenVZ container and runs Ubuntu 14.04
> The postgresql version is 9.3
> 3 other clusters on the same container are fine.
> We use a hardware RAID10 of SATA disks with a BBU (and writeBack mode)
Right offhand, I wonder whether the weak link in that isn't OpenVZ.
Virtualization technologies have a long and ugly reputation for not
providing strong filesystem-integrity guarantees.
regards, tom lane