WAL replay asking for very old WAL

Hello,

I have a 2 servers that are geographically separated be about 570KM which are connected via LAN extension. The slave at location 2 fell behind and I’ve had to restore it with pg_basebackup then replay my WAL files. Part way through the replay I get an error and then it starts asking for an old out of sequence WAL file that I no longer have. It continues to restore the WAL files in the sequence but continually askes for the old one, even after it has caught up to the live environment. I’ve tried it three separate times and it is always the same file name. There is un unlisten command that seems to trigger this to happen, but I’m not sure where it came from.

Any help is appreciated.

PostgreSQL 9.3.3

…

2016-03-06 00:25:45 UTC LOG:  restored log file "000000010000578800000038" from archive

2016-03-06 00:25:45 UTC FATAL:  the database system is starting up

2016-03-06 00:25:45 UTC FATAL:  the database system is starting up

2016-03-06 00:25:45 UTC FATAL:  the database system is starting up

2016-03-06 00:25:45 UTC FATAL:  the database system is starting up

2016-03-06 00:25:45 UTC FATAL:  the database system is starting up

2016-03-06 00:25:47 UTC LOG:  restored log file "000000010000578800000039" from archive

2016-03-06 00:25:48 UTC LOG:  restored log file "00000001000057880000003A" from archive

2016-03-06 00:25:49 UTC FATAL:  the database system is starting up

2016-03-06 00:25:50 UTC LOG:  restored log file "00000001000057880000003B" from archive

2016-03-06 00:25:50 UTC FATAL:  the database system is starting up

2016-03-06 00:25:51 UTC LOG:  restored log file "00000001000057880000003C" from archive

2016-03-06 00:25:53 UTC LOG:  restored log file "00000001000057880000003D" from archive

2016-03-06 00:25:53 UTC LOG:  consistent recovery state reached at 5788/3DCB5EF8

2016-03-06 00:25:53 UTC LOG:  database system is ready to accept read only connections

2016-03-06 00:25:54 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:54 UTC STATEMENT:  unlisten *

2016-03-06 00:25:54 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:54 UTC STATEMENT:  unlisten *

2016-03-06 00:25:54 UTC LOG:  restored log file "00000001000057880000003E" from archive

2016-03-06 00:25:54 UTC ERROR:  requested WAL segment 00000001000054540000007E has already been removed

2016-03-06 00:25:55 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:55 UTC STATEMENT:  unlisten *

2016-03-06 00:25:55 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:55 UTC STATEMENT:  unlisten *

2016-03-06 00:25:55 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:55 UTC STATEMENT:  unlisten *

2016-03-06 00:25:55 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:25:55 UTC STATEMENT:  unlisten *

2016-03-06 00:25:56 UTC LOG:  restored log file "00000001000057880000003F" from archive

2016-03-06 00:25:57 UTC LOG:  restored log file "000000010000578800000040" from archive

2016-03-06 00:25:58 UTC LOG:  restored log file "000000010000578800000041" from archive

2016-03-06 00:26:00 UTC ERROR:  requested WAL segment 00000001000054540000007E has already been removed

2016-03-06 00:26:00 UTC LOG:  restored log file "000000010000578800000042" from archive

2016-03-06 00:26:00 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:26:00 UTC STATEMENT:  unlisten *

2016-03-06 00:26:01 UTC LOG:  restored log file "000000010000578800000043" from archive

2016-03-06 00:26:02 UTC LOG:  restored log file "000000010000578800000044" from archive

2016-03-06 00:26:03 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:26:03 UTC STATEMENT:  unlisten *

2016-03-06 00:26:03 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:26:03 UTC STATEMENT:  unlisten *

2016-03-06 00:26:03 UTC ERROR:  cannot execute UNLISTEN during recovery

2016-03-06 00:26:03 UTC STATEMENT:  unlisten *

2016-03-06 00:26:03 UTC LOG:  restored log file "000000010000578800000045" from archive

2016-03-06 00:26:04 UTC LOG:  restored log file "000000010000578800000046" from archive

2016-03-06 00:26:05 UTC ERROR:  requested WAL segment 00000001000054540000007E has already been removed

2016-03-06 00:26:06 UTC LOG:  restored log file "000000010000578800000047" from archive

2016-03-06 00:26:07 UTC LOG:  restored log file "000000010000578800000048" from archive

2016-03-06 00:26:07 UTC LOG:  restored log file "000000010000578800000049" from archive

2016-03-06 00:26:09 UTC LOG:  restored log file "00000001000057880000004A" from archive

2016-03-06 00:26:10 UTC ERROR:  requested WAL segment 00000001000054540000007E has already been removed….