Re: Excessive growth of pg_attribute and other system tables

Steve Crawford wrote:

>On Monday 21 March 2005 11:40 am, Tom Lane wrote:
>
>
>>"Matthew T. O'Connor" <matthew@zeut.net> writes:
>>
>>
>>>I believe this discrepancy has to do with the fact that ANALYZE
>>>can return some very bogus values for reltuples, where as vacuum
>>>always returns an accurate count.  I'm not sure how to best
>>>handle this.
>>>
>>>
>>I think 8.0's ANALYZE will do a better estimation job ... at least,
>>Manfred Koizar rewrote the sampling algorithm in hopes of making it
>>more robust.
>>
>>However, given that there are 9334 tuples in 82282 pages, I'd say
>>that autovacuum has already failed Steve rather badly :-(.  There
>>shouldn't be more than a couple hundred pages given that number of
>>rows.  Perhaps the FSM settings are too small?
>>
>>
>
>Results time. FSM settings were too small but the real problem seems
>to be that pg_autovacuum isn't getting the job done.
>
>

Yeah, it certainly looks that way...

>I ran VACUUM VERBOSE and set postgresql.conf as follows (roughly 2.5
>times the amounts suggested by VACUUM VERBOSE results):
>max_fsm_pages = 500000
>max_fsm_relations = 2500
>
>I restarted the server last Friday night and manually did a vacuum
>full and reindex of each user and system table in every database on
>Monday. The zzz.pg_attribute table's file size dropped from over
>600MB to less than 2MB.
>
>

Wow, that is some serious bloat.

>It's now three days later and that table has already increased to
>29MB. Processes accessing this database do create many temporary
>tables so the exact count in pg_attribute varies a bit from time to
>time but always hovers around 9500. I just did a manual VACUUM FULL
>on pg_attribute and it's back to 1.3MB.
>
>

Depending on several factors the "steady state" size of pg_attribute may
be several times larger than it's size right after a vacuum full.  The
problem is not that it may be 29M, but rather that it continues to grow.

>Upon completion of the vacuum, I restarted the pg_autovacuum daemon.
>Following are the autovacuum log entries related to zzz.pg_attribute.
>Note the growth of reltuples from 9532.000000 to 184720.000000 in
>that time (what is a millionth of a tuple, anyway?) and the lack of
>any vacuums performed.
>
>

I assume that pg_attribute didn't actually grow to 184720 tuples?

>28 12:12 PM]   table name: zzz."pg_catalog"."pg_attribute"
>28 12:12 PM]      relid: 1249;   relisshared: 0
>28 12:12 PM]      reltuples: 9532.000000;  relpages: 157
>28 12:12 PM]      curr_analyze_count: 176294; curr_vacuum_count: 15447
>28 12:12 PM]      last_analyze_count: 176294; last_vacuum_count: 15447
>28 12:12 PM]      analyze_threshold: 10032; vacuum_threshold: 20064
>28 12:12 PM] added table: zzz."pg_catalog"."pg_attribute"
>--
>28 03:42 PM] Performing: ANALYZE "pg_catalog"."pg_attribute"
>28 03:42 PM]   table name: zzz."pg_catalog"."pg_attribute"
>28 03:42 PM]      relid: 1249;   relisshared: 0
>28 03:42 PM]      reltuples: 20270.000000;  relpages: 336
>28 03:42 PM]      curr_analyze_count: 186374; curr_vacuum_count: 16329
>28 03:42 PM]      last_analyze_count: 186374; last_vacuum_count: 15447
>28 03:42 PM]      analyze_threshold: 20770; vacuum_threshold: 20064
>--
>28 10:59 PM] Performing: ANALYZE "pg_catalog"."pg_attribute"
>28 10:59 PM]   table name: zzz."pg_catalog"."pg_attribute"
>28 10:59 PM]      relid: 1249;   relisshared: 0
>28 10:59 PM]      reltuples: 42591.000000;  relpages: 706
>28 10:59 PM]      curr_analyze_count: 207254; curr_vacuum_count: 18156
>28 10:59 PM]      last_analyze_count: 207254; last_vacuum_count: 15447
>28 10:59 PM]      analyze_threshold: 43091; vacuum_threshold: 20064
>--
>29 02:03 PM] Performing: ANALYZE "pg_catalog"."pg_attribute"
>29 02:03 PM]   table name: zzz."pg_catalog"."pg_attribute"
>29 02:03 PM]      relid: 1249;   relisshared: 0
>29 02:03 PM]      reltuples: 89464.000000;  relpages: 1483
>29 02:03 PM]      curr_analyze_count: 250664; curr_vacuum_count: 21999
>29 02:03 PM]      last_analyze_count: 250664; last_vacuum_count: 15447
>29 02:03 PM]      analyze_threshold: 89964; vacuum_threshold: 20064
>--
>30 09:20 PM] Performing: ANALYZE "pg_catalog"."pg_attribute"
>30 09:20 PM]   table name: zzz."pg_catalog"."pg_attribute"
>30 09:20 PM]      relid: 1249;   relisshared: 0
>30 09:20 PM]      reltuples: 184720.000000;  relpages: 3062
>30 09:20 PM]      curr_analyze_count: 340791; curr_vacuum_count: 29886
>30 09:20 PM]      last_analyze_count: 340791; last_vacuum_count: 15447
>30 09:20 PM]      analyze_threshold: 185220; vacuum_threshold: 20064
>
>Should I abandon pg_autovacuum and just do periodic VACUUM ANALYZEs of
>everything? Should I use settings other than the defaults for
>pg_autovacuum? Are temporary tables evil and their use to be avoided?
>
>Just checked and zzz.pg_attribute is 50% larger than it was when I did
>the VACUUM FULL at the start of this email.
>

I don't think you should you abandon pg_autovacuum, but I do think you
should run periodic vacuum commands from cron until this is resolved.

The 1st thing you should do it change the pg_autovacuum settings.  The
default values are very (far too) conservative for a lot of people.
Most people seem to have success with settings like:
 pg_autovacuum -v 300 -V 0.1  -a 200 -A 0.1
Give that a try and see if it helps.

 From pg_autovacuum's prospective, from 3/28 12:12 PM to 3/30 09:20 PM
there have been 29886 - 15447 = 14439 deletes + update commands against
the pg_attribute table.  Once it sees 20064 deletes + updates it will
perform a vacuum.  If there have been more during that time, we need to
figure out why pg_autovacuum is not seeing them.

The thing I don't understand in your numbers is why reltuples is
constantly increasing.  When pg_autovacuum is reporting a reltuples of
184720,  how many tuples does pg_attribute really have?   This might be
related to the new reltuples estimation code that Tom was talking about
recently, if so, 8.0.2 might help.

Anyone else have some insight here?

Matt