Hello,
We're using streaming replication. Our technique for spinning up a db
slave is this:
rsync from master (gross copy)
pg_start_backup() on server
rsync from master (correct copy)
pg_stop_backup()
drop in recovery.conf into slave directory
enable hot_standby in slave conf
start slave
After starting the slave, I'm getting this error every 5 seconds in the log:
ERROR: requested WAL segment 0000000100000E2200000005 has already been
removed
But I can connect to the DB and make queries and new records are
definitely streaming in.
I thought I just didn't have enough wal segments, so I bumped up the
number on the master and restarted the process. It just finished the
second time and the exact same error message is in the logs again (same
wal segment number).
When I ran pg_start_backup() and pg_stop_backup(), the output was:
pg_start_backup
-----------------
E27/3100A200
pg_stop_backup
----------------
E28/7D357950
The backup_label file looked like this:
START WAL LOCATION: E27/3100A200 (file 0000000100000E2700000031)
CHECKPOINT LOCATION: E27/31C9C740
BACKUP METHOD: pg_start_backup
BACKUP FROM: master
START TIME: 2016-04-02 12:34:25 PDT
LABEL: clone
During the rsync it copied
pg_xlog/0000000100000E25000000F1
to
pg_xlog/0000000100000E2800000071
So I'm confused: why is the E22 wal being requested? It seems to predate
the backup by a lot.
Does the slave really contain all the data? If not, how can I tell what
is missing (and why is it accepting streaming data if it's missing
something)?
One more piece of the puzzle that may or may not be relevant:
The current master used to be streaming replication slave. The original
master had a disk failure and so we switched one of the backup slaves
into a master. We've replaced the disk on the original server and we're
now trying to make it a streaming replication slave. This is the part
that's failing. If I do rough estimates of how fast the Exx number is
incrementing and compute backwards, E22 seems like about the time of the
original disk failure, give or take.
Thanks,
David