Обсуждение: Too many WAL archive files

Поиск
Список
Период
Сортировка

Too many WAL archive files

От
Keith Ouellette
Дата:

My company is using PostgreSQL 9.1 for one of our applications. We have it set up replicating between two sites using WAL and Pacemaker to manage the cluster and failover. I have noticed that archive folder is growing very large 16G out of the 19G that the data directory is in total. We have the wal_keep_segments = 1000 set in the postgresql.conf file.

 

I thought that meant to keep the last 1000 files in the event that it was needed to catch up after falling behind. I am noticing that I have 12000 files in the archive directory. I know that includes the .backup files as well, but that is a smaller portion of the files in that directory.

 

Is there something that I am missing? Below is the replication section of the configuration file, if that helps.

 

Any guidance would be greatly appreciated,

 

Keith

 

#------------------------------------------------------------------------------
# REPLICATION
#------------------------------------------------------------------------------

# - Master Server -

# These settings are ignored on a standby server

max_wal_senders = 2        # max number of walsender processes
                # (change requires restart)
#wal_sender_delay = 1s        # walsender cycle time, 1-10000 milliseconds
wal_keep_segments = 1000        # in logfile segments, 16MB each; 0 disables
#vacuum_defer_cleanup_age = 0    # number of xacts by which cleanup is delayed
#replication_timeout = 60s    # in milliseconds; 0 disables
#synchronous_standby_names = ''    # standby servers that provide sync rep
                # comma-separated list of application_name
                # from standby(s); '*' = all

# - Standby Servers -

# These settings are ignored on a master server

hot_standby = on            # "on" allows queries during recovery
                    # (change requires restart)
max_standby_archive_delay = -1    # max delay before canceling queries
                    # when reading WAL from archive;
                    # -1 allows indefinite delay
max_standby_streaming_delay = -1    # max delay before canceling queries
                    # when reading streaming WAL;
                    # -1 allows indefinite delay
wal_receiver_status_interval = 2s    # send replies at least this often
                    # 0 disables
hot_standby_feedback = on        # send info from standby to prevent
                    # query conflicts

Re: Too many WAL archive files

От
Matheus de Oliveira
Дата:

On Sat, Sep 14, 2013 at 11:19 AM, Keith Ouellette <Keith.Ouellette@airgas.com> wrote:

My company is using PostgreSQL 9.1 for one of our applications. We have it set up replicating between two sites using WAL and Pacemaker to manage the cluster and failover. I have noticed that archive folder is growing very large 16G out of the 19G that the data directory is in total. We have the wal_keep_segments = 1000 set in the postgresql.conf file.

 

I thought that meant to keep the last 1000 files in the event that it was needed to catch up after falling behind.


No. PostgreSQL will always keep at least (can be a little more) wal_keep_segments files in any situation. So, wal_keep_segments is set to 1000, and each wal file has 16MB, doing the math it means PostgreSQL will use at least 15.625GB (~16GB) for wal files, it is what you have. If you don't have enough space for this, you should set keep wal_keep_segments to a lower value.

I am noticing that I have 12000 files in the archive directory. I know that includes the .backup files as well, but that is a smaller portion of the files in that directory.


Unless you have a lot of .backup files, there is no way 12000 wal file will use only 16GB, it would be 187.5GB, so this number seems wrong.

Also check if you have set up archiving and if it is working, because if archive_command fails, PostgreSQL will keep the "failed on archive" files on pg_xlog path and keep trying it.

Best regards,
-- 
Matheus de Oliveira
Analista de Banco de Dados
Dextra Sistemas - MPS.Br nível F!
www.dextra.com.br/postgres

Re: Too many WAL archive files

От
Jorge Torralba
Дата:
You should consider using the cleanup command in your recovery.conf file, this will make sure that wal files no longer needed by the secondary server are eliminated.

If you need the wal files for PITR, yo could also setup your archive command to rchive the wal files to two different locations. a shared location where the secondary can read if it needs to catch up and a keep location which is used by the primary just for PITR which you could then clean up after each base backup.

In case you ask, if this is a unix box, you can separatethe two commands for archive_command with  && for example

comand 1 && command 2

JT


On Sat, Sep 14, 2013 at 8:55 AM, Matheus de Oliveira <matioli.matheus@gmail.com> wrote:

On Sat, Sep 14, 2013 at 11:19 AM, Keith Ouellette <Keith.Ouellette@airgas.com> wrote:

My company is using PostgreSQL 9.1 for one of our applications. We have it set up replicating between two sites using WAL and Pacemaker to manage the cluster and failover. I have noticed that archive folder is growing very large 16G out of the 19G that the data directory is in total. We have the wal_keep_segments = 1000 set in the postgresql.conf file.

 

I thought that meant to keep the last 1000 files in the event that it was needed to catch up after falling behind.


No. PostgreSQL will always keep at least (can be a little more) wal_keep_segments files in any situation. So, wal_keep_segments is set to 1000, and each wal file has 16MB, doing the math it means PostgreSQL will use at least 15.625GB (~16GB) for wal files, it is what you have. If you don't have enough space for this, you should set keep wal_keep_segments to a lower value.

I am noticing that I have 12000 files in the archive directory. I know that includes the .backup files as well, but that is a smaller portion of the files in that directory.


Unless you have a lot of .backup files, there is no way 12000 wal file will use only 16GB, it would be 187.5GB, so this number seems wrong.

Also check if you have set up archiving and if it is working, because if archive_command fails, PostgreSQL will keep the "failed on archive" files on pg_xlog path and keep trying it.

Best regards,
-- 
Matheus de Oliveira
Analista de Banco de Dados
Dextra Sistemas - MPS.Br nível F!
www.dextra.com.br/postgres




--
Thanks,

Jorge Torralba
----------------------------

Note: This communication may contain privileged or other confidential information. If you are not the intended recipient, please do not print, copy, retransmit, disseminate or otherwise use the information. Please indicate to the sender that you have received this email in error and delete the copy you received. Thank You.

Re: Too many WAL archive files

От
Keith Ouellette
Дата:

Okay, I guess it makes sense that I am at 16GB based on your explanation below. I do not have a space constraint, so having 16GB is not the problem. The problem is that we have experienced connectivity issues in the past where replication would not startup. I would do a "manual" sync using rsync for the data directory. That includes the archive directory. I tried to exclude the archive directory, but when restarting PostgreSQL, I would get the archive missing errors in the startup log and PostgreSQL would not start. I guess I could reduce the number from 1000 (which looks like about 55 days with an average of 18 files being created each day) to a lower number. We picked 1000 as this is a new application for us and we did not know how fast the WAL files would fill.  If I do make that change, will the excess files be automatically deleted or do I have to do that manually?

 

Thanks,

Keith

 


From: Matheus de Oliveira [matioli.matheus@gmail.com]
Sent: Saturday, September 14, 2013 11:55 AM
To: Keith Ouellette
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Too many WAL archive files


On Sat, Sep 14, 2013 at 11:19 AM, Keith Ouellette <Keith.Ouellette@airgas.com> wrote:

My company is using PostgreSQL 9.1 for one of our applications. We have it set up replicating between two sites using WAL and Pacemaker to manage the cluster and failover. I have noticed that archive folder is growing very large 16G out of the 19G that the data directory is in total. We have the wal_keep_segments = 1000 set in the postgresql.conf file.

 

I thought that meant to keep the last 1000 files in the event that it was needed to catch up after falling behind.


No. PostgreSQL will always keep at least (can be a little more) wal_keep_segments files in any situation. So, wal_keep_segments is set to 1000, and each wal file has 16MB, doing the math it means PostgreSQL will use at least 15.625GB (~16GB) for wal files, it is what you have. If you don't have enough space for this, you should set keep wal_keep_segments to a lower value.

I am noticing that I have 12000 files in the archive directory. I know that includes the .backup files as well, but that is a smaller portion of the files in that directory.


Unless you have a lot of .backup files, there is no way 12000 wal file will use only 16GB, it would be 187.5GB, so this number seems wrong.

Also check if you have set up archiving and if it is working, because if archive_command fails, PostgreSQL will keep the "failed on archive" files on pg_xlog path and keep trying it.

Best regards,
-- 
Matheus de Oliveira
Analista de Banco de Dados
Dextra Sistemas - MPS.Br nível F!
www.dextra.com.br/postgres

Re: Too many WAL archive files

От
pavan95
Дата:
Hi Matheus de Oliveira,

It is the same case with me too. The archive files are being generated to a
vast extent of 2 GB per hour. And the wal_keep_segments parameter is set 0. 

And we are not following any replication practices. The size of the database
is 24GB. Daily 48GB of archive logs are being generated and this abnormal
behaviour started from December 28'th 2017.

The set of DML's which were ran before and after December 28'th 2017 are the
same. So why these many archive logs are being generated? Those are causing
a severe disk space problem.

And FYI autovacuum was set to on. But when I tried doing vacuumdb for this
database almost 1GB of archive logs are getting generated per 1 sec. 

It seems to be very strange, any idea of why it is generating?(to prevent
transaction Id wraparound failures? If yes how long it will generate?)

I'm facing this issue from December 28'th 2017. Please suggest how to cope
with it. Thanks in Advance.

Regards,
Pavan



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-admin-f2076596.html