Обсуждение: pg_archivecleanup - Increase time files are deleted

Поиск
Список
Период
Сортировка

pg_archivecleanup - Increase time files are deleted

От
Patrick B
Дата:
Hi guys,

I've got one MASTER and three SLAVES servers, running PostgreSQL 9.2.

I had a problem with one of the masters, and because the wal_files aren't kept for a long time, I had to re-sync all the DB.
If the wal_files were being kept longer than they are now, I wouldn't have to re-sync (pg_basebackup) all the DB.

So, I'd like to have the wal_files stored for 24h, and then the pg_archivecleanup could do its job and delete the files....


How can I increase that?


These are my wal_files now: (got from a slave)

-rw------- 1 postgres postgres 16777216 Jun 26 21:04 00000002000013CA000000CA
-rw------- 1 postgres postgres 16777216 Jun 26 21:05 00000002000013CA000000CB
-rw------- 1 postgres postgres 16777216 Jun 26 21:05 00000002000013CA000000CC
-rw------- 1 postgres postgres 16777216 Jun 26 21:06 00000002000013CA000000CD
-rw------- 1 postgres postgres 16777216 Jun 26 21:07 00000002000013CA000000CE
-rw------- 1 postgres postgres 16777216 Jun 26 21:08 00000002000013CA000000CF
-rw------- 1 postgres postgres 16777216 Jun 26 21:09 00000002000013CA000000D0
-rw------- 1 postgres postgres 16777216 Jun 26 21:09 00000002000013CA000000D1
-rw------- 1 postgres postgres 16777216 Jun 26 21:10 00000002000013CA000000D2
-rw------- 1 postgres postgres 16777216 Jun 26 21:10 00000002000013CA000000D3
-rw------- 1 postgres postgres 16777216 Jun 26 21:11 00000002000013CA000000D4
-rw------- 1 postgres postgres 16777216 Jun 26 21:12 00000002000013CA000000D5

Re: pg_archivecleanup - Increase time files are deleted

От
Melvin Davidson
Дата:


On Sun, Jun 26, 2016 at 5:14 PM, Patrick B <patrickbakerbr@gmail.com> wrote:
Hi guys,

I've got one MASTER and three SLAVES servers, running PostgreSQL 9.2.

I had a problem with one of the masters, and because the wal_files aren't kept for a long time, I had to re-sync all the DB.
If the wal_files were being kept longer than they are now, I wouldn't have to re-sync (pg_basebackup) all the DB.

So, I'd like to have the wal_files stored for 24h, and then the pg_archivecleanup could do its job and delete the files....


How can I increase that?


These are my wal_files now: (got from a slave)

-rw------- 1 postgres postgres 16777216 Jun 26 21:04 00000002000013CA000000CA
-rw------- 1 postgres postgres 16777216 Jun 26 21:05 00000002000013CA000000CB
-rw------- 1 postgres postgres 16777216 Jun 26 21:05 00000002000013CA000000CC
-rw------- 1 postgres postgres 16777216 Jun 26 21:06 00000002000013CA000000CD
-rw------- 1 postgres postgres 16777216 Jun 26 21:07 00000002000013CA000000CE
-rw------- 1 postgres postgres 16777216 Jun 26 21:08 00000002000013CA000000CF
-rw------- 1 postgres postgres 16777216 Jun 26 21:09 00000002000013CA000000D0
-rw------- 1 postgres postgres 16777216 Jun 26 21:09 00000002000013CA000000D1
-rw------- 1 postgres postgres 16777216 Jun 26 21:10 00000002000013CA000000D2
-rw------- 1 postgres postgres 16777216 Jun 26 21:10 00000002000013CA000000D3
-rw------- 1 postgres postgres 16777216 Jun 26 21:11 00000002000013CA000000D4
-rw------- 1 postgres postgres 16777216 Jun 26 21:12 00000002000013CA000000D5


>...I'd like to have the wal_files stored for 24h, and then the pg_archivecleanup could do its job and delete the files..

You cannot do that, but you can change the wal_keep_segments on the master and reload the postgresql.conf.

https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS

Please, in the future, remember to state your PostgreSQL version and O/S as options can change.

--
Melvin Davidson
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.

Re: pg_archivecleanup - Increase time files are deleted

От
Patrick B
Дата:


>...I'd like to have the wal_files stored for 24h, and then the pg_archivecleanup could do its job and delete the files..

You cannot do that, but you can change the wal_keep_segments on the master and reload the postgresql.conf.

https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS

ok.. my current wal_keep_segments is:

wal_keep_segments = 256

That means there will be at least 256 files before postgres delete them?

 


Please, in the future, remember to state your PostgreSQL version and O/S as options can change.


Ok.. my mistake.. I'm running PostgreSQL 9.2 in a Centos 6.7 64 bits. 

Re: pg_archivecleanup - Increase time files are deleted

От
Melvin Davidson
Дата:

On Sun, Jun 26, 2016 at 5:33 PM, Patrick B <patrickbakerbr@gmail.com> wrote:


>...I'd like to have the wal_files stored for 24h, and then the pg_archivecleanup could do its job and delete the files..

You cannot do that, but you can change the wal_keep_segments on the master and reload the postgresql.conf.

https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS

ok.. my current wal_keep_segments is:

wal_keep_segments = 256

That means there will be at least 256 files before postgres delete them?

 


Please, in the future, remember to state your PostgreSQL version and O/S as options can change.


Ok.. my mistake.. I'm running PostgreSQL 9.2 in a Centos 6.7 64 bits. 


>That means there will be at least 256 files before postgres delete them?

Yes!
--
Melvin Davidson
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.

Re: pg_archivecleanup - Increase time files are deleted

От
Patrick B
Дата:
Apparently I can't do that via Postgres.

So I've made a bash script to do the work for me.

The script will only delete the wal_files older than 24h ( -mmin +1440 )


* It has been tested and it's working.

recovery.conf:
archive_cleanup_command = 'exec nice -n 19 ionice -c 2 -n 7 ../archivecleaup_mv.bash -d "../wal_archive" "%r"'

archivecleaup_mv.bash:
#!/bin/bash -eu

declare -r -x PATH='/usr/local/bin:/usr/bin:/bin';

ARCHIVEDIR='/var/lib/pgsql/9.2/wal_archive/'
CHKPOINT=$(find $ARCHIVEDIR -type f -mmin +1440 -name '00*' -printf '%f\n' | sort -r | head -1)
#cd $ARCHIVEDIR

exec "/usr/pgsql-9.2/bin/pg_archivecleanup" $ARCHIVEDIR $CHKPOINT;
find $ARCHIVEDIR -type f -mmin +1440 -a -name '00*' -a ! -newer $CHKPOINT -delete

Re: pg_archivecleanup - Increase time files are deleted

От
Ian Barwick
Дата:
On 06/27/2016 06:33 AM, Patrick B wrote:
>>> ...I'd like to have the wal_files stored for 24h, and then the
>> pg_archivecleanup could do its job and delete the files..
>>
>>
>> *You cannot do that, but you can change the wal_keep_segments
>> <https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS>
>> on the master and reload the postgresql.conf.*
>> *https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS
>> <https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS>*
>>
>
> ok.. my current wal_keep_segments is:
>
> wal_keep_segments = 256
> That means there will be at least 256 files before postgres delete them?
>>
>> Please, in the future, remember to state your *PostgreSQL version and O/S*
>> as options can change.
>>
>>
> Ok.. my mistake.. I'm running PostgreSQL 9.2 in a Centos 6.7 64 bits.

FYI PostgreSQL 9.4 and later provide "replication slots", which will ensure that all WAL
files are kept until no longer required by standbys:

   https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS

This removes the need for guesswork on how much WAL to retain, though you do then need to be
careful that all standbys are actually consuming WAL otherwise files will be retained for ever
(or until disk space runs out, whichever comes first).

Regards

Ian Barwick


--
  Ian Barwick                   http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services


Re: pg_archivecleanup - Increase time files are deleted

От
Melvin Davidson
Дата:


On Sun, Jun 26, 2016 at 7:39 PM, Ian Barwick <ian@2ndquadrant.com> wrote:
On 06/27/2016 06:33 AM, Patrick B wrote:
...I'd like to have the wal_files stored for 24h, and then the
pg_archivecleanup could do its job and delete the files..


*You cannot do that, but you can change the wal_keep_segments
<https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS>
on the master and reload the postgresql.conf.*
*https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS
<https://www.postgresql.org/docs/9.4/static/runtime-config-replication.html#GUC-WAL-KEEP-SEGMENTS>*


ok.. my current wal_keep_segments is:

wal_keep_segments = 256
That means there will be at least 256 files before postgres delete them?

Please, in the future, remember to state your *PostgreSQL version and O/S*
as options can change.


Ok.. my mistake.. I'm running PostgreSQL 9.2 in a Centos 6.7 64 bits.

FYI PostgreSQL 9.4 and later provide "replication slots", which will ensure that all WAL
files are kept until no longer required by standbys:

  https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS

This removes the need for guesswork on how much WAL to retain, though you do then need to be
careful that all standbys are actually consuming WAL otherwise files will be retained for ever
(or until disk space runs out, whichever comes first).

Regards

Ian Barwick


--
 Ian Barwick                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Patrick

>So I've made a bash script to do the work for me.
>The script will only delete the wal_files older than 24h ( -mmin +1440 )

It may be working, but that is not the supported method for wal file retention. The correct thing to do is just
incrtease the wal_keep_segments as per the documentation https://www.postgresql.org/docs/9.2/static/runtime-config-replication.html

>FYI PostgreSQL 9.4 and later provide "replication slots", which will ensure that all WAL
>files are kept until no longer required by standbys:

replication slots will not work, as you are on 9.2.
 
Melvin Davidson
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.

Re: pg_archivecleanup - Increase time files are deleted

От
Michael Paquier
Дата:
On Mon, Jun 27, 2016 at 6:42 AM, Melvin Davidson <melvin6925@gmail.com> wrote:
> >That means there will be at least 256 files before postgres delete them?
>
> Yes!

Not exactly. To be more precise, for an instance generating WAL, the
decision regarding the number of WAL segments to remove or recycled is
done at the creation of a checkpoint, for a standby that's when a
restart point is created. See where KeepLogSeg() in xlog.c.
--
Michael