Обсуждение: autovacuum ignore tables

Sriram Dandapani wrote:
>
> If I were to specify in the pg_autovacuum catalog that certain high
> volume partitioned tables(that get dropped daily) be ignored, then
> when autovacuum finishes, will it update the transaction id wraparound
> counter (this way, I can get autovacuum to finish quickly )
>
> OR
>
> Will I still need to periodically do vacuumdb –a to take care of the
> wraparound problem.
>

You don't need to do a manual vacuumdb -a since autovacuum will do this
once it decides you are getting too close to the wraparound point. I
believe this has been improved in the upcoming 8.2 release where
autovacuum no longer need to vacuum the whole database at once, rather
XID wraparound is now tracked on a per table basis.

The only issue I have with autovacuum is the fact that I have to briefly
stop/restart postgres every couple of days, which kills autovacuum and
it has no memory of previous work done. I work with several databases
with partitioned tables having high daily volume. Dropping partitioned
tables locks out jdbc inserts and the drop command itself goes into a
WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent
access,restart postgres,drop tables and update pg_hba to allow
access(Crazy, but I have no choice because Postgres deadlocks on drop
child tables while inserts happen on the parent)

-----Original Message-----
From: Matthew T. O'Connor [mailto:matthew@zeut.net]
Sent: Thursday, September 28, 2006 9:22 PM
To: Sriram Dandapani
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] autovacuum ignore tables

Sriram Dandapani wrote:
>
> If I were to specify in the pg_autovacuum catalog that certain high
> volume partitioned tables(that get dropped daily) be ignored, then
> when autovacuum finishes, will it update the transaction id wraparound
> counter (this way, I can get autovacuum to finish quickly )
>
> OR
>
> Will I still need to periodically do vacuumdb -a to take care of the
> wraparound problem.
>

You don't need to do a manual vacuumdb -a since autovacuum will do this
once it decides you are getting too close to the wraparound point. I
believe this has been improved in the upcoming 8.2 release where
autovacuum no longer need to vacuum the whole database at once, rather
XID wraparound is now tracked on a per table basis.

Sriram Dandapani wrote:
> The only issue I have with autovacuum is the fact that I have to briefly
> stop/restart postgres every couple of days, which kills autovacuum and
> it has no memory of previous work done. I work with several databases
> with partitioned tables having high daily volume. Dropping partitioned
> tables locks out jdbc inserts and the drop command itself goes into a
> WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent
> access,restart postgres,drop tables and update pg_hba to allow
> access(Crazy, but I have no choice because Postgres deadlocks on drop
> child tables while inserts happen on the parent)

Sounds like a fairly serious problem, but I'll let someone else talk to
that.  As for the autovac issue, as of 8.1 and autovac being integrated
into core, this isn't true.  It does remember where it was, at least as
long as you don't have the GUC var reset_stats_on_restart (something
like that) set.

Matt

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> The only issue I have with autovacuum is the fact that I have to briefly
> stop/restart postgres every couple of days, which kills autovacuum and
> it has no memory of previous work done.

As already stated, this isn't true as of 8.1.

> I work with several databases
> with partitioned tables having high daily volume. Dropping partitioned
> tables locks out jdbc inserts and the drop command itself goes into a
> WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent
> access,restart postgres,drop tables and update pg_hba to allow
> access(Crazy, but I have no choice because Postgres deadlocks on drop
> child tables while inserts happen on the parent)

It sounds to me like you have a problem with failing to commit
transactions promptly.  The DROP will wait for existing transactions to
release their locks on the doomed table, but its lock request will block
any incoming transactions that try to acquire new locks on the table.
So basically you should see a hiccup of length equal to the longest
normal transaction using that table.  In a well-designed concurrent
system that should not be a problem.

It might be worth doing the DROP as

    begin;
    lock table parent_table;
    drop table child_table;
    commit;

so that SELECTs on the parent do simply block and don't risk getting
errors from trying to access a just-dropped child table.

            regards, tom lane

I tried your code with the lock table followed by drop.

The lock table goes into a wait state AND all INSERTS (even new ones
coming in) go into a wait state.

Ps -ef | grep waiting    shows the function call (that does the lock and
drop) in wait state as well as all INSERTS.

All jdbc inserts are committed cleanly. When the commit is done..the
autocommit for the connection is set to true (to flush out any open
transactions)



-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Friday, September 29, 2006 7:45 AM
To: Sriram Dandapani
Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] autovacuum ignore tables

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> The only issue I have with autovacuum is the fact that I have to
briefly
> stop/restart postgres every couple of days, which kills autovacuum and
> it has no memory of previous work done.

As already stated, this isn't true as of 8.1.

> I work with several databases
> with partitioned tables having high daily volume. Dropping partitioned
> tables locks out jdbc inserts and the drop command itself goes into a
> WAIT state. Hence, I have to stop postgres,update pg_hba.conf to
prevent
> access,restart postgres,drop tables and update pg_hba to allow
> access(Crazy, but I have no choice because Postgres deadlocks on drop
> child tables while inserts happen on the parent)

It sounds to me like you have a problem with failing to commit
transactions promptly.  The DROP will wait for existing transactions to
release their locks on the doomed table, but its lock request will block
any incoming transactions that try to acquire new locks on the table.
So basically you should see a hiccup of length equal to the longest
normal transaction using that table.  In a well-designed concurrent
system that should not be a problem.

It might be worth doing the DROP as

    begin;
    lock table parent_table;
    drop table child_table;
    commit;

so that SELECTs on the parent do simply block and don't risk getting
errors from trying to access a just-dropped child table.

            regards, tom lane

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> I tried your code with the lock table followed by drop.
> The lock table goes into a wait state AND all INSERTS (even new ones
> coming in) go into a wait state.

Well, it's waiting for something.  Try looking in pg_locks to see which
transaction is blocking it.

            regards, tom lane

I see RowExclusiveLocks for all the jdbc inserts

Real question is: Why do the INSERTS go into wait state as soon as the
lock table statement is issued on the parent?

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Friday, September 29, 2006 11:10 AM
To: Sriram Dandapani
Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] autovacuum ignore tables

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> I tried your code with the lock table followed by drop.
> The lock table goes into a wait state AND all INSERTS (even new ones
> coming in) go into a wait state.

Well, it's waiting for something.  Try looking in pg_locks to see which
transaction is blocking it.

            regards, tom lane

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> Real question is: Why do the INSERTS go into wait state as soon as the
> lock table statement is issued on the parent?

If you were just inserting directly into other child tables, a lock on
either the parent or the target child table shouldn't affect them.
I wonder if you are using conditional rules to redirect the inserts,
and the rules include a reference to the target table?  If that's the
case, you really need to drop the relevant rule before you remove the
child table, anyway.

            regards, tom lane

The jdbc inserts go into the main parent table and check constraints
redirect them to child tables.

If I were to drop the rule, that would immediately affect the inserts
and they would go the the parent table. And I cannot lock the parent due
to the problem I mentioned earlier.


-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Friday, September 29, 2006 2:38 PM
To: Sriram Dandapani
Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] autovacuum ignore tables

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> Real question is: Why do the INSERTS go into wait state as soon as the
> lock table statement is issued on the parent?

If you were just inserting directly into other child tables, a lock on
either the parent or the target child table shouldn't affect them.
I wonder if you are using conditional rules to redirect the inserts,
and the rules include a reference to the target table?  If that's the
case, you really need to drop the relevant rule before you remove the
child table, anyway.

            regards, tom lane

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> The jdbc inserts go into the main parent table and check constraints
> redirect them to child tables.

A check constraint can't redirect anything.  Are you saying that you use
a trigger to try to insert the row into *each* child table, relying on
the constraints to make all except one insert fail?  Ugh.  You'd do a
lot better to duplicate the partitioning-rule knowledge in the trigger,
and do only one insert that should succeed --- no wasted cycles, and no
need for a subtransaction.

            regards, tom lane

I meant to say...rules are used on the parent table to redirect to
child. Just like the way the postgres 8.1 documentation has it.

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Friday, September 29, 2006 2:48 PM
To: Sriram Dandapani
Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] autovacuum ignore tables

"Sriram Dandapani" <sdandapani@counterpane.com> writes:
> The jdbc inserts go into the main parent table and check constraints
> redirect them to child tables.

A check constraint can't redirect anything.  Are you saying that you use
a trigger to try to insert the row into *each* child table, relying on
the constraints to make all except one insert fail?  Ugh.  You'd do a
lot better to duplicate the partitioning-rule knowledge in the trigger,
and do only one insert that should succeed --- no wasted cycles, and no
need for a subtransaction.

            regards, tom lane