Обсуждение: autovacuum can not remove dead tuples
Hi,all
We meet a problem in postgreql 9.5.5.When we run autovacuum or vacuum manually,it always goes " dead row version
cannotbe removed yet".I think it is about Visable MAP,how can we skip the VM,and do the complete vacuum?
INFO: "ela_elifeassist_product": found 0 removable, 187435 nonremovable row versions in 23981 out of 23981 pages
DETAIL: 185659 dead row versions cannot be removed yet.
There were 2993 unused item pointers.
Skipped 0 pages due to buffer pins.
0 pages are entirely empty.
CPU 0.14s/0.10u sec elapsed 0.25 sec.
INFO: vacuuming "pg_toast.pg_toast_231114"
INFO: index "pg_toast_231114_index" now contains 35 row versions in 2 pages
DETAIL: 0 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: "pg_toast_231114": found 0 removable, 35 nonremovable row versions in 6 out of 6 pages
DETAIL: 0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
Skipped 0 pages due to buffer pins.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM
====== End: Tue Apr 3 17:23:31 CST 2018
[postgres:5543@ela] [04-03.17:27:09]=# select pid,state,substr(query,0,100) from pg_stat_activity where query ~
'ela_elifeassist_product';
pid | state | substr
-------+--------+-----------------------------------------------------------------------------------------------------
38726 | active | select pid,state,substr(query,0,100) from pg_stat_activity where query ~ 'ela_elifeassist_product'
10482 | idle | SELECT +
| | t.tablename, +
| | indexname, +
| | c.reltuples AS num_rows, +
| | pg_size_pretty(pg_relation_
26023 | idle | select * from pg_stat_activity where query like '%ela_elifeassist_product%';
40180 | active | autovacuum: VACUUM public.ela_elifeassist_product
(4 rows)
Time: 2.919 ms
[postgres:5543@ela] [04-03.17:27:13]=# select a.locktype,a.pid,a.relation,a.mode,a.granted,b.relname from pg_locks
a,pg_classb where a.relation=b.oid and relname='ela_elifeassist_product';
locktype | pid | relation | mode | granted | relname
----------+-------+----------+--------------------------+---------+-------------------------
relation | 40180 | 231114 | ShareUpdateExclusiveLock | t | ela_elifeassist_product
(1 row)
Time: 1.056 ms
[postgres:5543@ela] [04-03.17:27:57]=# select relname,relpages,relallvisible from pg_class where relname
='ela_elifeassist_product';
relname | relpages | relallvisible
-------------------------+----------+---------------
ela_elifeassist_product | 23981 | 18
(1 row)
Time: 8.679 ms
********************************************************************************************************************************
The information in this email is confidential and may be legally privileged. If you have received this email in error
orare not the intended recipient, please immediately notify the sender and delete this message from your computer. Any
use,distribution, or copying of this email other than by the intended recipient is strictly prohibited. All messages
sentto and from us may be monitored to ensure compliance with internal policies and to protect our business.
Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed,
orcontain viruses. Anyone who communicates with us by email is taken to accept these risks.
收发邮件者请注意：
本邮件含涉密信息，请保守秘密，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
********************************************************************************************************************************
Hello Have you any replica with hot_standby_feedback=on ? Long transactions on such replicas will prevent vacuum. Another importantoption is vacuum_defer_cleanup_age on master. regards, Sergei
Hello And an additional question: > [postgres:5543@ela] [04-03.17:27:09]=# select pid,state,substr(query,0,100) from pg_stat_activity where query ~ 'ela_elifeassist_product'; Have you another long transactions related not to this table? Any long transaction will prevent vacuum. Vacuum can only delete rows older than the oldest transaction regardless relations.For example, transaction begin; select * from tablename1; -- sleep 10 min select * from another_tablename; commit; We must leave some dead rows in another_tablename for this 10 minutes because we do not know which rows are required in anopen transaction later. regards Sergei.
I think we have foud the cause,after we killed the session that state is idle in transation ,dead tuples can be
removed.Iknow that in PG9.6 we can use DISABLE_PAGE_SKIPPING,but 9.5 can`t .
But we still don’t know how this idle in transaction session comes.maybe it is produced by DBeaver,we need to have a
test.
DBID |dbname| username| program | ip address | begi_time |state |Query
17482 | ela | ela | | 2018-04-03 01:00:00 | active | select
collect_customer_job();
17482 | ela | ela | DBeaver 3.7.7 - Main | xxx.xxx.xx.xxx | 2018-04-03 08:51:48 | idle in transaction
|SELECT current_database(), current_schema(),session_user
17482 | ela | dbm DBeaver 3.7.7 - Main |xxx.xxx.xx.xxx | 2018-04-03 16:20:02.| idle in transaction |
SELECTcurrent_database(), current_schema(),session_user
-----邮件原件-----
发件人: Sergei Kornilov [mailto:sk@zsrv.org]
发送时间: 2018年4月3日 22:19
收件人: 石勇虎 <SHIYONGHU651@pingan.com.cn>; pgsql-bugs@postgresql.org
主题: Re: autovacuum can not remove dead tuples
Hello
And an additional question:
> [postgres:5543@ela] [04-03.17:27:09]=# select
> pid,state,substr(query,0,100) from pg_stat_activity where query ~
> 'ela_elifeassist_product' ;
Have you another long transactions related not to this table?
Any long transaction will prevent vacuum. Vacuum can only delete rows older than the oldest transaction regardless
relations.For example, transaction begin; select * from tablename1;
-- sleep 10 min
select * from another_tablename;
commit;
We must leave some dead rows in another_tablename for this 10 minutes because we do not know which rows are required in
anopen transaction later.
regards Sergei.
********************************************************************************************************************************
The information in this email is confidential and may be legally privileged. If you have received this email in error
orare not the intended recipient, please immediately notify the sender and delete this message from your computer. Any
use,distribution, or copying of this email other than by the intended recipient is strictly prohibited. All messages
sentto and from us may be monitored to ensure compliance with internal policies and to protect our business.
Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed,
orcontain viruses. Anyone who communicates with us by email is taken to accept these risks.
收发邮件者请注意：
本邮件含涉密信息，请保守秘密，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
********************************************************************************************************************************
Hello
Yes, any session in transaction will prevent vacuum some rows. Regardless DISABLE_PAGE_SKIPPING or any other option,
thisis not bug. Vacuum should not remove any deleted row which can be accessible by any running transaction.
You can use idle_in_transaction_session_timeout to prevent long idle in transaction state. But i use simple crontab
likethis:
*/10 * * * * psql -d postgres -tx -c "SELECT pg_terminate_backend(pid),now(),now()-xact_start as duration,* from
pg_stat_activitywhere (now() - pg_stat_activity.xact_start) > '1 hour'::interval and usename NOT IN ('postgres') and
state<>'idle'"
to terminate any long running transaction. Time interval changed by database workload, for web requests can be 5min.
regards, Sergei