I am facing an issue with postgres replication between my primary and DR site. I have the following setup,
1. I am trying to replicate LVM level sanpshot on SAN which does a block level replication.
2. OS Details : RHEL 7.1 kernel 3.10
3. Postgres Version : ( 9.6)
The steps performed:
1. Stop all the containers running on the OS.
2. Stop the SAN level replication.
3. Switch over to the replicated site.
4. Start the containers
Here the postgres container fails with the blow error which looks like data corruption.
========
LOG: database system was interrupted; last known up at 2017-04-28 15:58:45 UTC
LOG: invalid magic number 7270 in log segment 000000010000000000000001, offset 0
LOG: invalid primary checkpoint record
LOG: invalid magic number 7270 in log segment 000000010000000000000001, offset 0
LOG: invalid secondary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 18) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
=======
I have tried the graceful shutdown of the microservices but still the replication fails. Strange issues id i have other instance of postgres (9.4.1 )which runs absolutely fine. Could someone please provide some advice?