Обсуждение: Standby is not removing restored WAL segments

Поиск
Список
Период
Сортировка

Standby is not removing restored WAL segments

От
Alexey Klyukin
Дата:
Greetings,

We've got a 9.3.5 DB running in a standby mode for a fairly large DB
(500GB) with a busy WAL traffic (couple of GBs per hour) and it
occasionally 'forgets' to remove the segments it restored.

The checkpoint_segments is set to 128, and usually we observe around
270 segments accumulated, but at the time it happens our check
triggers at around 2K segments. The manual checkpoint command takes
ages to complete there,  the fast shutdown is very slow (around 10
minutes, usually less than 1 minute) and the WAL receiver process is
also unable to run for some reason.

The only way to make this host delete WAL files is to restart . The
particularly notable restart point right after the shutdown shows
quite a number of removed files and buffers written (the shared
buffers is set to 8GB on this system):

2014-09-04 14:39:33.376 CEST,,,22354,,537a4553.5752,88217,,2014-05-19
19:54:27 CEST,,0,LOG,00000,"restartpoint complete: wrote 332473
buffers (31.7%); 0 transaction log file(s) added, 1237 removed, 6
recycled; write=9.745 s, sync=680.314 s, total=694.447 s; sync
files=499
, longest=37.774 s, average=1.363 s",,,,,,,,,""

If we leave the host running, this restartpoint never happens.

The only difference I can come up with from the other databases that
do not show this behavior is that the host is running with
max_standby_streaming_delay and max_standby_archive_delay set to -1,
but at the time we observed the problem no queries were running on it
at all.

The problem occurs rarely, but steadily, around once every 3 months.
During this time the PostgreSQL has been upgraded from 9.0 to 9.3,
which did not solve the issue.

Any clues on how can we debug and diagnose the problem further to come
up with a proper bug report, if it is a bug, or are we missing
something in the configuration that causes this?


Regards,
--
Alexey Klyukin

Re: Standby is not removing restored WAL segments

От
bricklen
Дата:

On Fri, Sep 5, 2014 at 12:33 AM, Alexey Klyukin <alexk@hintbits.com> wrote:
We've got a 9.3.5 DB running in a standby mode for a fairly large DB
(500GB) with a busy WAL traffic (couple of GBs per hour) and it
occasionally 'forgets' to remove the segments it restored.

A complete shot in the dark, but do you take backups from that server, and if so, does the backup script "pause" replication via "pg_xlog_replay_pause()"? If so, "select pg_is_xlog_replay_paused();" should return true, and executing "select pg_xlog_replay_resume()" will un-pause replication.

Again, that's only a WAG.

Re: Standby is not removing restored WAL segments

От
Alexey Klyukin
Дата:
On Mon, Sep 8, 2014 at 3:52 AM, bricklen <bricklen@gmail.com> wrote:
>
> On Fri, Sep 5, 2014 at 12:33 AM, Alexey Klyukin <alexk@hintbits.com> wrote:
>>
>> We've got a 9.3.5 DB running in a standby mode for a fairly large DB
>> (500GB) with a busy WAL traffic (couple of GBs per hour) and it
>> occasionally 'forgets' to remove the segments it restored.
>
>
> A complete shot in the dark, but do you take backups from that server, and
> if so, does the backup script "pause" replication via
> "pg_xlog_replay_pause()"? If so, "select pg_is_xlog_replay_paused();" should
> return true, and executing "select pg_xlog_replay_resume()" will un-pause
> replication.



>
> Again, that's only a WAG.
>

Thanks. we do take DB backups from this server, but not from this
cluster on it, so it's not a backup interference.

--
Regards,
Alexey Klyukin

Re: Standby is not removing restored WAL segments

От
Jerry Sievers
Дата:
Alexey Klyukin <alexk@hintbits.com> writes:

> On Mon, Sep 8, 2014 at 3:52 AM, bricklen <bricklen@gmail.com> wrote:
>
>>
>> On Fri, Sep 5, 2014 at 12:33 AM, Alexey Klyukin <alexk@hintbits.com> wrote:
>>>
>>> We've got a 9.3.5 DB running in a standby mode for a fairly large DB
>>> (500GB) with a busy WAL traffic (couple of GBs per hour) and it
>>> occasionally 'forgets' to remove the segments it restored.
>>
>>
>> A complete shot in the dark, but do you take backups from that server, and
>> if so, does the backup script "pause" replication via
>> "pg_xlog_replay_pause()"? If so, "select pg_is_xlog_replay_paused();" should
>> return true, and executing "select pg_xlog_replay_resume()" will un-pause
>> replication.
>
>
>
>>
>> Again, that's only a WAG.
>>
>
> Thanks. we do take DB backups from this server, but not from this
> cluster on it, so it's not a backup interference.

Or...  Are your backups being taken using rsync or any non-pg_basebackup
method *and* you are copying unnecessary WALs to the target new standby
server, which then lay around forever as artifacts of not having been
omitted initially?

>
> --
> Regards,
> Alexey Klyukin

--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net
p: 312.241.7800

Re: Standby is not removing restored WAL segments

От
Eduardo Morras
Дата:
On Fri, 5 Sep 2014 09:33:57 +0200
Alexey Klyukin <alexk@hintbits.com> wrote:

> Greetings,
>
> We've got a 9.3.5 DB running in a standby mode for a fairly large DB
> (500GB) with a busy WAL traffic (couple of GBs per hour) and it
> occasionally 'forgets' to remove the segments it restored.
>
> The checkpoint_segments is set to 128, and usually we observe around
> 270 segments accumulated, but at the time it happens our check
> triggers at around 2K segments. The manual checkpoint command takes
> ages to complete there,  the fast shutdown is very slow (around 10
> minutes, usually less than 1 minute) and the WAL receiver process is
> also unable to run for some reason.
>
> The only way to make this host delete WAL files is to restart . The
> particularly notable restart point right after the shutdown shows
> quite a number of removed files and buffers written (the shared
> buffers is set to 8GB on this system):
>
> 2014-09-04 14:39:33.376 CEST,,,22354,,537a4553.5752,88217,,2014-05-19
> 19:54:27 CEST,,0,LOG,00000,"restartpoint complete: wrote 332473
> buffers (31.7%); 0 transaction log file(s) added, 1237 removed, 6
> recycled; write=9.745 s, sync=680.314 s, total=694.447 s; sync
> files=499
> , longest=37.774 s, average=1.363 s",,,,,,,,,""
>
> If we leave the host running, this restartpoint never happens.
>
> The only difference I can come up with from the other databases that
> do not show this behavior is that the host is running with
> max_standby_streaming_delay and max_standby_archive_delay set to -1,
> but at the time we observed the problem no queries were running on it
> at all.
>
> The problem occurs rarely, but steadily, around once every 3 months.
> During this time the PostgreSQL has been upgraded from 9.0 to 9.3,
> which did not solve the issue.
>

Perhaps, the delete of wal files occurs before, in filesystem time, the wal file is closed by filesystem, and delete
returns"error file still open". 

> Any clues on how can we debug and diagnose the problem further to come
> up with a proper bug report, if it is a bug, or are we missing
> something in the configuration that causes this?
>
>
> Regards,
> --
> Alexey Klyukin
>
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin


---   ---
Eduardo Morras <emorrasg@yahoo.es>

Re: Standby is not removing restored WAL segments

От
Alexey Klyukin
Дата:
On Mon, Sep 8, 2014 at 7:06 PM, Jerry Sievers <gsievers19@comcast.net> wrote:

> Or...  Are your backups being taken using rsync or any non-pg_basebackup
> method *and* you are copying unnecessary WALs to the target new standby
> server, which then lay around forever as artifacts of not having been
> omitted initially?

That contradicts the fact that the files were removed after the restart.

--
Regards,
Alexey Klyukin

Re: Standby is not removing restored WAL segments

От
Guillaume Lelarge
Дата:
Hi,

2014-09-05 9:33 GMT+02:00 Alexey Klyukin <alexk@hintbits.com>:
Greetings,

We've got a 9.3.5 DB running in a standby mode for a fairly large DB
(500GB) with a busy WAL traffic (couple of GBs per hour) and it
occasionally 'forgets' to remove the segments it restored.

The checkpoint_segments is set to 128, and usually we observe around
270 segments accumulated, but at the time it happens our check
triggers at around 2K segments. The manual checkpoint command takes
ages to complete there,  the fast shutdown is very slow (around 10
minutes, usually less than 1 minute) and the WAL receiver process is
also unable to run for some reason.

The only way to make this host delete WAL files is to restart . The
particularly notable restart point right after the shutdown shows
quite a number of removed files and buffers written (the shared
buffers is set to 8GB on this system):

2014-09-04 14:39:33.376 CEST,,,22354,,537a4553.5752,88217,,2014-05-19
19:54:27 CEST,,0,LOG,00000,"restartpoint complete: wrote 332473
buffers (31.7%); 0 transaction log file(s) added, 1237 removed, 6
recycled; write=9.745 s, sync=680.314 s, total=694.447 s; sync
files=499
, longest=37.774 s, average=1.363 s",,,,,,,,,""

If we leave the host running, this restartpoint never happens.

The only difference I can come up with from the other databases that
do not show this behavior is that the host is running with
max_standby_streaming_delay and max_standby_archive_delay set to -1,
but at the time we observed the problem no queries were running on it
at all.

The problem occurs rarely, but steadily, around once every 3 months.
During this time the PostgreSQL has been upgraded from 9.0 to 9.3,
which did not solve the issue.

Any clues on how can we debug and diagnose the problem further to come
up with a proper bug report, if it is a bug, or are we missing
something in the configuration that causes this?


I have no direct answer for you, but we seem to have the same issue for two of our customers. We are on 9.2.8 on one of them. Do you know if you have the .ready related files in the archive_status directory? are they old WAL files? can you tell us their names?

We're still investigating the issue. Not that it's a real issue, but it's still weird. And we'd like to understand what's happening.


--

Re: Standby is not removing restored WAL segments

От
Alexey Klyukin
Дата:
On Mon, Sep 15, 2014 at 5:33 PM, Guillaume Lelarge
<guillaume@lelarge.info> wrote:

> I have no direct answer for you, but we seem to have the same issue for two
> of our customers. We are on 9.2.8 on one of them. Do you know if you have
> the .ready related files in the archive_status directory? are they old WAL
> files? can you tell us their names?

We don't have their names anymore, but the stalled WAL segments were
from the same
timeline as the up-to-date ones and were removed after we did a restart.

As for the ready files, there are quite a lot of them:


pg_xlog/archive_status$ ls -la
total 68
drwx------ 2 postgres postgres 28672 Sep 16 12:46 .
drwxr-xr-x 3 postgres postgres 20480 Sep 16 12:41 ..
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D3.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D4.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D5.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D6.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D7.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D8.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D9.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DA.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DB.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DC.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000ED.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000000.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000001.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000002.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000003.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000004.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000005.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000006.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000007.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000008.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000009.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000010.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000011.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000012.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000013.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000014.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000015.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000016.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000017.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000018.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000019.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000020.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000021.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000022.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000023.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000024.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000025.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000026.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000027.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000028.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000029.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000030.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000031.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000032.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000033.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000034.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000035.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000036.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000037.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000038.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000039.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C180000003F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000045.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000046.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000047.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000048.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000049.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000050.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000051.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000052.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000053.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000054.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000055.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000056.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000057.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000058.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000059.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000060.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000061.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000062.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000063.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000064.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000065.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000066.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000067.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000068.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000069.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000070.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000071.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000072.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000073.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000074.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000075.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000076.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000077.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000078.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000079.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000080.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000081.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000082.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000083.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000084.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000085.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000086.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000087.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000088.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000089.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000090.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000091.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000092.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000093.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000094.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000095.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000096.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000097.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000098.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000099.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000CA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000045.ready
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002D.done
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002E.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A120000002F.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000030.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000031.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000032.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000033.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000034.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000035.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000036.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000037.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000038.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000039.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003A.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003B.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003C.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003D.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003E.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003F.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000040.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000041.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000042.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000043.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000044.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000045.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000046.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000047.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000048.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000049.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004A.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004B.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004C.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004D.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004E.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004F.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000050.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000051.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000052.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000053.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000054.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000055.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000056.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000057.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000058.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A1200000059.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005A.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005B.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005C.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005D.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005E.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005F.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000060.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000061.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000062.done
-rw------- 1 postgres postgres     0 Sep 16 12:46 0000000300007A1200000063.done
-rw------- 1 postgres postgres     0 Apr 10 16:49 00000003.history.done
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000ED.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000000.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000001.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000002.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000003.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000004.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000005.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000006.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000007.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000008.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000009.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000010.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000011.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000012.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000013.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000014.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000015.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000016.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000017.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000018.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000019.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000020.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000021.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000022.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000023.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000024.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000025.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000026.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000027.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000028.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000029.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000030.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000031.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000032.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000033.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000034.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000035.ready

I think most of them were generated to the moment right near the
timeline switch, likely
related to rebuilding of the master (and this replica) due to 9.3
upgrade or for some
other purposes.  They do not correspond to the time when the database
was experiencing
issues described in the first post.

--
Regards,
Alexey Klyukin

Re: Standby is not removing restored WAL segments

От
Alexey Klyukin
Дата:
On Tue, Sep 16, 2014 at 2:20 PM, Adalkonda Harshad
<adalkondaharshad@gmail.com> wrote:
>

> I suspect there is a problem while setting the archive_command that resulting towards accumulating WAL .ready file in
pg_xlog/archive_statusdirectory itself. 

I think archive_command has no effect on the non-primary server, and
the problem described above is happening on standby.
It's hard to suspect the restore_command as well, because the WAL
segments are copied to the pg_xlog as a result of the restore_command,
and therefore, it's working.

Re: Standby is not removing restored WAL segments

От
Guillaume Lelarge
Дата:
2014-09-16 15:03 GMT+02:00 Alexey Klyukin <alexk@hintbits.com>:
On Tue, Sep 16, 2014 at 2:20 PM, Adalkonda Harshad
<adalkondaharshad@gmail.com> wrote:
>

> I suspect there is a problem while setting the archive_command that resulting towards accumulating WAL .ready file in pg_xlog/archive_status directory itself.

I think archive_command has no effect on the non-primary server, and
the problem described above is happening on standby.
It's hard to suspect the restore_command as well, because the WAL
segments are copied to the pg_xlog as a result of the restore_command,
and therefore, it's working.

I agree with Alexey. There's nothing wrong with our archive_command. Not sure what's wrong though.


--

Re: Standby is not removing restored WAL segments

От
Guillaume Lelarge
Дата:
2014-09-16 12:57 GMT+02:00 Alexey Klyukin <alexk@hintbits.com>:
On Mon, Sep 15, 2014 at 5:33 PM, Guillaume Lelarge
<guillaume@lelarge.info> wrote:

> I have no direct answer for you, but we seem to have the same issue for two
> of our customers. We are on 9.2.8 on one of them. Do you know if you have
> the .ready related files in the archive_status directory? are they old WAL
> files? can you tell us their names?

We don't have their names anymore, but the stalled WAL segments were
from the same
timeline as the up-to-date ones and were removed after we did a restart.

As for the ready files, there are quite a lot of them:


pg_xlog/archive_status$ ls -la
total 68
drwx------ 2 postgres postgres 28672 Sep 16 12:46 .
drwxr-xr-x 3 postgres postgres 20480 Sep 16 12:41 ..
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D3.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D4.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D5.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D6.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D7.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D8.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D9.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DA.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DB.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DC.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000ED.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000000.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000001.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000002.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000003.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000004.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000005.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000006.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000007.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000008.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000009.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000010.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000011.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000012.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000013.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000014.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000015.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000016.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000017.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000018.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000019.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000020.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000021.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000022.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000023.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000024.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000025.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000026.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000027.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000028.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000029.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000030.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000031.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000032.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000033.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000034.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000035.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000036.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000037.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000038.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000039.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C180000003F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000045.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000046.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000047.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000048.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000049.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000050.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000051.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000052.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000053.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000054.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000055.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000056.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000057.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000058.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000059.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000060.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000061.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000062.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000063.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000064.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000065.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000066.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000067.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000068.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000069.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000070.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000071.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000072.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000073.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000074.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000075.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000076.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000077.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000078.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000079.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000080.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000081.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000082.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000083.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000084.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000085.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000086.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000087.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000088.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000089.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000090.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000091.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000092.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000093.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000094.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000095.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000096.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000097.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000098.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000099.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000CA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000045.ready
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002D.done
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002E.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A120000002F.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000030.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000031.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000032.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000033.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000034.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000035.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000036.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000037.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000038.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000039.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003A.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003B.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003C.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003D.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003E.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003F.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000040.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000041.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000042.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000043.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000044.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000045.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000046.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000047.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000048.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000049.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004A.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004B.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004C.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004D.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004E.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004F.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000050.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000051.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000052.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000053.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000054.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000055.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000056.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000057.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000058.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A1200000059.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005A.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005B.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005C.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005D.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005E.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005F.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000060.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000061.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000062.done
-rw------- 1 postgres postgres     0 Sep 16 12:46 0000000300007A1200000063.done
-rw------- 1 postgres postgres     0 Apr 10 16:49 00000003.history.done
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000ED.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000000.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000001.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000002.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000003.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000004.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000005.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000006.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000007.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000008.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000009.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000010.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000011.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000012.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000013.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000014.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000015.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000016.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000017.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000018.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000019.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000020.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000021.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000022.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000023.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000024.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000025.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000026.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000027.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000028.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000029.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000030.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000031.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000032.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000033.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000034.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000035.ready

I think most of them were generated to the moment right near the
timeline switch, likely
related to rebuilding of the master (and this replica) due to 9.3
upgrade or for some
other purposes.  They do not correspond to the time when the database
was experiencing
issues described in the first post.


Wow, that's a lot of .ready fles. Our customer's server only has six of them. We also have the related WAL files. Do you?

And if we delete the .ready files, they come back as soon as a restart point is done IIRC.


--

Re: Standby is not removing restored WAL segments

От
Adalkonda Harshad
Дата:

On 16-09-2014 16:27, Alexey Klyukin wrote:
On Mon, Sep 15, 2014 at 5:33 PM, Guillaume Lelarge
<guillaume@lelarge.info> wrote:


I have no direct answer for you, but we seem to have the same issue for two
of our customers. We are on 9.2.8 on one of them. Do you know if you have
the .ready related files in the archive_status directory? are they old WAL
files? can you tell us their names?

We don't have their names anymore, but the stalled WAL segments were
from the same
timeline as the up-to-date ones and were removed after we did a restart.

As for the ready files, there are quite a lot of them:


pg_xlog/archive_status$ ls -la
total 68
drwx------ 2 postgres postgres 28672 Sep 16 12:46 .
drwxr-xr-x 3 postgres postgres 20480 Sep 16 12:41 ..
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D3.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D4.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D5.ready
-rw------- 1 postgres postgres     0 Apr 10 16:47 0000000100006C17000000D6.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D7.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D8.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000D9.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DA.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DB.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DC.ready
-rw------- 1 postgres postgres     0 Apr 10 16:56 0000000100006C17000000DD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000DF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000E9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000ED.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000EF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000F9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C17000000FF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000000.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000001.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000002.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000003.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000004.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000005.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000006.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000007.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000008.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000009.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000000F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000010.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000011.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000012.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000013.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000014.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000015.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000016.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000017.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000018.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000019.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000001F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000020.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000021.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000022.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000023.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000024.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000025.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000026.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000027.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000028.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000029.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000002F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000030.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000031.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000032.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000033.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000034.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000035.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000036.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000037.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000038.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C1800000039.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:02 0000000100006C180000003E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C180000003F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000100006C1800000045.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000046.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000047.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000048.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000049.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C180000004F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000050.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000051.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000052.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000053.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000054.ready
-rw------- 1 postgres postgres     0 Apr 10 17:12 0000000100006C1800000055.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000056.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000057.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000058.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000059.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000005F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000060.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000061.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000062.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000063.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000064.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000065.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000066.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000067.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000068.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000069.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C180000006F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000070.ready
-rw------- 1 postgres postgres     0 Apr 10 17:17 0000000100006C1800000071.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000072.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000073.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000074.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000075.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000076.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000077.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000078.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000079.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C180000007F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000080.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000081.ready
-rw------- 1 postgres postgres     0 Apr 10 17:22 0000000100006C1800000082.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000083.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000084.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000085.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000086.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000087.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000088.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000089.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C180000008F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000090.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000091.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000092.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000093.ready
-rw------- 1 postgres postgres     0 Apr 10 17:27 0000000100006C1800000094.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000095.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000096.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000097.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000098.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C1800000099.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009A.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009B.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009C.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009D.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009E.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C180000009F.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:33 0000000100006C18000000A3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000A9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:40 0000000100006C18000000AC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000AF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000B9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BB.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BC.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BD.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BE.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000BF.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C0.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C1.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C2.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C3.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C4.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C5.ready
-rw------- 1 postgres postgres     0 Apr 10 17:49 0000000100006C18000000C6.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C7.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C8.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000C9.ready
-rw------- 1 postgres postgres     0 Apr 10 17:55 0000000100006C18000000CA.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000040.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000041.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000042.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000043.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000044.ready
-rw------- 1 postgres postgres     0 Apr 10 17:07 0000000200006C1800000045.ready
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002D.done
-rw------- 1 postgres postgres     0 Sep 16 12:27 0000000300007A120000002E.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A120000002F.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000030.done
-rw------- 1 postgres postgres     0 Sep 16 12:28 0000000300007A1200000031.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000032.done
-rw------- 1 postgres postgres     0 Sep 16 12:29 0000000300007A1200000033.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000034.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000035.done
-rw------- 1 postgres postgres     0 Sep 16 12:30 0000000300007A1200000036.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000037.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000038.done
-rw------- 1 postgres postgres     0 Sep 16 12:31 0000000300007A1200000039.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003A.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003B.done
-rw------- 1 postgres postgres     0 Sep 16 12:32 0000000300007A120000003C.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003D.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003E.done
-rw------- 1 postgres postgres     0 Sep 16 12:33 0000000300007A120000003F.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000040.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000041.done
-rw------- 1 postgres postgres     0 Sep 16 12:34 0000000300007A1200000042.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000043.done
-rw------- 1 postgres postgres     0 Sep 16 12:35 0000000300007A1200000044.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000045.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000046.done
-rw------- 1 postgres postgres     0 Sep 16 12:36 0000000300007A1200000047.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000048.done
-rw------- 1 postgres postgres     0 Sep 16 12:37 0000000300007A1200000049.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004A.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004B.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004C.done
-rw------- 1 postgres postgres     0 Sep 16 12:38 0000000300007A120000004D.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004E.done
-rw------- 1 postgres postgres     0 Sep 16 12:39 0000000300007A120000004F.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000050.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000051.done
-rw------- 1 postgres postgres     0 Sep 16 12:40 0000000300007A1200000052.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000053.done
-rw------- 1 postgres postgres     0 Sep 16 12:41 0000000300007A1200000054.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000055.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000056.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000057.done
-rw------- 1 postgres postgres     0 Sep 16 12:42 0000000300007A1200000058.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A1200000059.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005A.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005B.done
-rw------- 1 postgres postgres     0 Sep 16 12:43 0000000300007A120000005C.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005D.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005E.done
-rw------- 1 postgres postgres     0 Sep 16 12:44 0000000300007A120000005F.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000060.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000061.done
-rw------- 1 postgres postgres     0 Sep 16 12:45 0000000300007A1200000062.done
-rw------- 1 postgres postgres     0 Sep 16 12:46 0000000300007A1200000063.done
-rw------- 1 postgres postgres     0 Apr 10 16:49 00000003.history.done
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000B9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000BF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000C9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000CF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000D9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000DF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000E9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000ED.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000EF.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F0.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F1.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F2.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F3.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F4.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F5.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F6.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F7.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F8.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000F9.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FA.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FB.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FC.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FD.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF4000000FE.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000000.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000001.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000002.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000003.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000004.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000005.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000006.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000007.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000008.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000009.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000000F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000010.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000011.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000012.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000013.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000014.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000015.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000016.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000017.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000018.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000019.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000001F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000020.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000021.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000022.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000023.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000024.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000025.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000026.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000027.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000028.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000029.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002A.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002B.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002C.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002D.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002E.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF50000002F.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000030.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000031.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000032.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000033.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000034.ready
-rw------- 1 postgres postgres     0 Apr  9 13:13 0000000800006BF500000035.ready

I think most of them were generated to the moment right near the
timeline switch, likely
related to rebuilding of the master (and this replica) due to 9.3
upgrade or for some
other purposes.  They do not correspond to the time when the database
was experiencing
issues described in the first post.
I suspect there is a problem while setting the archive_command that resulting towards accumulating WAL .ready file in pg_xlog/archive_status directory itself.
Please verify the archive_command that set at your end which should resolve the issue. One example  is provided for your reference.

Example:
    archive_command = 'cp %p /tmp/PostgreSQL/archive/%f'
--

Harshad Adalkonda
Database Administrator

Office: +919552687400/8400
http://www.shreeyansh.com

Вложения