Обсуждение: Update problem on large table

Поиск
Список
Период
Сортировка

Update problem on large table

От
felix
Дата:

Hello, 
I have a very large table that I'm not too fond of.  I'm revising the design now.

Up until now its been insert only, storing tracking codes from incoming webtraffic.

It has 8m rows
It appears to insert fine, but simple updates using psql are hanging.

update ONLY traffic_tracking2010 set src_content_type_id = 90 where id = 90322;

I am also now trying to remove the constraints, this also hangs.

alter table traffic_tracking2010 drop constraint traffic_tracking2010_src_content_type_id_fkey;

thanks in advance for any advice.


                                        Table "public.traffic_tracking2010"
       Column        |           Type           |                             Modifiers                             
---------------------+--------------------------+-------------------------------------------------------------------
 id                  | integer                  | not null default nextval('traffic_tracking2010_id_seq'::regclass)
 action_time         | timestamp with time zone | not null
 user_id             | integer                  | 
 content_type_id     | integer                  | 
 object_id           | integer                  | 
 action_type         | smallint                 | not null
 src_type            | smallint                 | 
 src_content_type_id | integer                  | 
 src_object_id       | integer                  | 
 http_referrer       | character varying(100)   | 
 search_term         | character varying(50)    | 
 remote_addr         | inet                     | not null
Indexes:
    "traffic_tracking2010_pkey" PRIMARY KEY, btree (id)
    "traffic_tracking2010_content_type_id" btree (content_type_id)
    "traffic_tracking2010_src_content_type_id" btree (src_content_type_id)
    "traffic_tracking2010_user_id" btree (user_id)
Foreign-key constraints:
    "traffic_tracking2010_content_type_id_fkey" FOREIGN KEY (content_type_id) REFERENCES django_content_type(id) DEFERRABLE INITIALLY DEFERRED
    "traffic_tracking2010_src_content_type_id_fkey" FOREIGN KEY (src_content_type_id) REFERENCES django_content_type(id) DEFERRABLE INITIALLY DEFERRED
    "traffic_tracking2010_user_id_fkey" FOREIGN KEY (user_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED


This is generated by Django's ORM.  

The hang may be do having other clients connected, though I have tried doing the update when I know all tracking inserts are stopped.
But the other client (the webapp) is still connected.

based on this:

ns=> ANALYZE traffic_tracking2010;
ANALYZE
ns=> SELECT relpages, reltuples FROM pg_class WHERE relname = 'traffic_tracking2010';
 relpages |  reltuples  
----------+-------------
    99037 | 8.38355e+06

and I did vacuum it

vacuum verbose traffic_tracking2010;
INFO:  vacuuming "public.traffic_tracking2010"
INFO:  scanned index "traffic_tracking2010_pkey" to remove 1057 row versions
DETAIL:  CPU 0.09s/0.37u sec elapsed 10.70 sec.
INFO:  scanned index "traffic_tracking2010_user_id" to remove 1057 row versions
DETAIL:  CPU 0.12s/0.30u sec elapsed 13.53 sec.
INFO:  scanned index "traffic_tracking2010_content_type_id" to remove 1057 row versions
DETAIL:  CPU 0.11s/0.28u sec elapsed 13.99 sec.
INFO:  scanned index "traffic_tracking2010_src_content_type_id" to remove 1057 row versions
DETAIL:  CPU 0.09s/0.26u sec elapsed 15.57 sec.
INFO:  "traffic_tracking2010": removed 1057 row versions in 535 pages
DETAIL:  CPU 0.01s/0.02u sec elapsed 2.83 sec.
INFO:  index "traffic_tracking2010_pkey" now contains 8315147 row versions in 22787 pages
DETAIL:  1057 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO:  index "traffic_tracking2010_user_id" now contains 8315147 row versions in 29006 pages
DETAIL:  1057 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO:  index "traffic_tracking2010_content_type_id" now contains 8315147 row versions in 28980 pages
DETAIL:  1057 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO:  index "traffic_tracking2010_src_content_type_id" now contains 8315147 row versions in 28978 pages
DETAIL:  1057 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO:  "traffic_tracking2010": found 336 removable, 8315147 nonremovable row versions in 99035 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
25953 pages contain useful free space.
0 pages are entirely empty.
CPU 0.78s/1.49u sec elapsed 100.43 sec.
INFO:  vacuuming "pg_toast.pg_toast_165961"
INFO:  index "pg_toast_165961_index" now contains 0 row versions in 1 pages
DETAIL:  0 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.01 sec.
INFO:  "pg_toast_165961": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages contain useful free space.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.06 sec.


Re: Update problem on large table

От
bricklen
Дата:
On Fri, Nov 26, 2010 at 6:22 AM, felix <crucialfelix@gmail.com> wrote:
>
> Hello,
> I have a very large table that I'm not too fond of.  I'm revising the design
> now.
> Up until now its been insert only, storing tracking codes from incoming
> webtraffic.
> It has 8m rows
> It appears to insert fine, but simple updates using psql are hanging.
> update ONLY traffic_tracking2010 set src_content_type_id = 90 where id =
> 90322;
> I am also now trying to remove the constraints, this also hangs.
> alter table traffic_tracking2010 drop constraint
> traffic_tracking2010_src_content_type_id_fkey;
> thanks in advance for any advice.

Try your update or alter and in another session, run the following
query and look at the "waiting" column. A "true" value means that that
transaction is blocked.

SELECT pg_stat_activity.datname, pg_stat_activity.procpid,
pg_stat_activity.usename, pg_stat_activity.current_query,
pg_stat_activity.waiting,
pg_stat_activity.query_start,pg_stat_activity.client_addr
FROM pg_stat_activity
WHERE ((pg_stat_activity.procpid <> pg_backend_pid())
AND (pg_stat_activity.current_query <> '<IDLE>'))
ORDER BY pg_stat_activity.query_start;

Re: Update problem on large table

От
felix
Дата:
Ok, I caught one : an update that is stuck in waiting.

the first one blocks the second one.

ns      |    5902 | nssql   | UPDATE "fastadder_fastadderstatus" SET "built" = false WHERE "fastadder_fastadderstatus"."service_id" = 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | f       | 2010-12-04 13:44:38.5228-05   | 127.0.0.1

 ns      |    7000 | nssql   | UPDATE "fastadder_fastadderstatus" SET "last_sent" = E'2010-12-04 13:50:51.452800', "sent" = true WHERE ("fastadder_fastadderstatus"."built" = true  AND "fastadder_fastadderstatus"."service_id" = 1 )                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | t       | 2010-12-04 13:50:51.4628-05   | 127.0.0.1

is it possible to release the lock and/or cancel the query ?  the process that initiated the first one is long ceased.






On Fri, Nov 26, 2010 at 6:02 PM, bricklen <bricklen@gmail.com> wrote:
No problem!

On Fri, Nov 26, 2010 at 8:34 AM, felix <crucialfelix@gmail.com> wrote:
> thanks !
> of course now, 2 hours later, the queries run fine.
> the first one was locked up for so long that I interrupted it.
> maybe that caused it to get blocked
> saved your query for future reference, thanks again !
> On Fri, Nov 26, 2010 at 5:00 PM, bricklen <bricklen@gmail.com> wrote:
>>
>> On Fri, Nov 26, 2010 at 6:22 AM, felix <crucialfelix@gmail.com> wrote:
>> >
>> > Hello,
>> > I have a very large table that I'm not too fond of.  I'm revising the
>> > design
>> > now.
>> > Up until now its been insert only, storing tracking codes from incoming
>> > webtraffic.
>> > It has 8m rows
>> > It appears to insert fine, but simple updates using psql are hanging.
>> > update ONLY traffic_tracking2010 set src_content_type_id = 90 where id =
>> > 90322;
>> > I am also now trying to remove the constraints, this also hangs.
>> > alter table traffic_tracking2010 drop constraint
>> > traffic_tracking2010_src_content_type_id_fkey;
>> > thanks in advance for any advice.
>>
>> Try your update or alter and in another session, run the following
>> query and look at the "waiting" column. A "true" value means that that
>> transaction is blocked.
>>
>> SELECT pg_stat_activity.datname, pg_stat_activity.procpid,
>> pg_stat_activity.usename, pg_stat_activity.current_query,
>> pg_stat_activity.waiting,
>> pg_stat_activity.query_start,pg_stat_activity.client_addr
>> FROM pg_stat_activity
>> WHERE ((pg_stat_activity.procpid <> pg_backend_pid())
>> AND (pg_stat_activity.current_query <> '<IDLE>'))
>> ORDER BY pg_stat_activity.query_start;
>>
>> --
>> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
>
>

Re: Update problem on large table

От
bricklen
Дата:
On Sat, Dec 4, 2010 at 11:45 AM, felix <crucialfelix@gmail.com> wrote:
> Ok, I caught one : an update that is stuck in waiting.
> the first one blocks the second one.
> ns      |    5902 | nssql   | UPDATE "fastadder_fastadderstatus" SET "built"
> = false WHERE "fastadder_fastadderstatus"."service_id" = 1

Not sure if anyone replied about killing your query, but you can do it like so:

select pg_cancel_backend(5902);  -- assuming 5902 is the pid of the
query you want canceled.

Re: Update problem on large table

От
Jon Nelson
Дата:
On Mon, Dec 6, 2010 at 1:46 PM, bricklen <bricklen@gmail.com> wrote:
> On Sat, Dec 4, 2010 at 11:45 AM, felix <crucialfelix@gmail.com> wrote:
>> Ok, I caught one : an update that is stuck in waiting.
>> the first one blocks the second one.
>> ns      |    5902 | nssql   | UPDATE "fastadder_fastadderstatus" SET "built"
>> = false WHERE "fastadder_fastadderstatus"."service_id" = 1
>
> Not sure if anyone replied about killing your query, but you can do it like so:
>
> select pg_cancel_backend(5902);  -- assuming 5902 is the pid of the
> query you want canceled.

How does this differ from just killing the pid?

--
Jon

Re: Update problem on large table

От
Josh Kupershmidt
Дата:
On Mon, Dec 6, 2010 at 2:48 PM, Jon Nelson <jnelson+pgsql@jamponi.net> wrote:
> On Mon, Dec 6, 2010 at 1:46 PM, bricklen <bricklen@gmail.com> wrote:
>> Not sure if anyone replied about killing your query, but you can do it like so:
>>
>> select pg_cancel_backend(5902);  -- assuming 5902 is the pid of the
>> query you want canceled.
>
> How does this differ from just killing the pid?

pg_cancel_backend(5902) does the same thing as:
  kill -SIGINT 5902

Josh

Re: Update problem on large table

От
Kenneth Marshall
Дата:
On Mon, Dec 06, 2010 at 03:24:31PM -0500, Josh Kupershmidt wrote:
> On Mon, Dec 6, 2010 at 2:48 PM, Jon Nelson <jnelson+pgsql@jamponi.net> wrote:
> > On Mon, Dec 6, 2010 at 1:46 PM, bricklen <bricklen@gmail.com> wrote:
> >> Not sure if anyone replied about killing your query, but you can do it like so:
> >>
> >> select pg_cancel_backend(5902); ?-- assuming 5902 is the pid of the
> >> query you want canceled.
> >
> > How does this differ from just killing the pid?
>
> pg_cancel_backend(5902) does the same thing as:
>   kill -SIGINT 5902
>
> Josh
>

Yes, but you can use it from within the database. The kill command
requires shell access to the backend.

Cheers,
Ken

Re: Update problem on large table

От
felix
Дата:

thanks for the replies !, 

but actually I did figure out how to kill it

but pb_cancel_backend didn't work.  here's some notes:

this has been hung for 5 days:
ns      |   32681 | nssql   | <IDLE> in transaction | f       | 2010-12-01 15

resulting in:  "fastadder_fastadderstatus": scanned 3000 of 58551 pages, containing 13587 live rows and 254709 dead rows; 
and resulting in general pandemonium 


you need to become the postgres superuser to use pg_cancel_backend:
 su postgres 
 psql

and then:

select pg_cancel_backend(32681);

but this does not kill the IDLE in transaction processes.
it returns true, but its still there

from the linux shell I tried:

pg_ctl kill INT 32681

but it still will not die

the docs for pg_ctl state:
"Use pb_ctl --help to see a list of supported signal names."

doing so does indeed tell me the names:

HUP INT QUIT ABRT TERM USR1 USR2

but nothing about them whatseover :)

throwing caution to the wind:

pg_ctl kill TERM 32681

and that did it

ran VACUUM and now performance has returned to normal.

lessons learned.

I guess as Josh says, pg_cancel_backend is the same as SIGINT, which also failed for me using pg_ctl.  
not sure why.  the hung transaction was doing something like update table set field = null where service_id = x



On Mon, Dec 6, 2010 at 9:26 PM, Kenneth Marshall <ktm@rice.edu> wrote:
On Mon, Dec 06, 2010 at 03:24:31PM -0500, Josh Kupershmidt wrote:
> On Mon, Dec 6, 2010 at 2:48 PM, Jon Nelson <jnelson+pgsql@jamponi.net> wrote:
> > On Mon, Dec 6, 2010 at 1:46 PM, bricklen <bricklen@gmail.com> wrote:
> >> Not sure if anyone replied about killing your query, but you can do it like so:
> >>
> >> select pg_cancel_backend(5902); ?-- assuming 5902 is the pid of the
> >> query you want canceled.
> >
> > How does this differ from just killing the pid?
>
> pg_cancel_backend(5902) does the same thing as:
>   kill -SIGINT 5902
>
> Josh
>

Yes, but you can use it from within the database. The kill command
requires shell access to the backend.

Cheers,
Ken

Re: Update problem on large table

От
Josh Kupershmidt
Дата:
On Mon, Dec 6, 2010 at 4:31 PM, felix <crucialfelix@gmail.com> wrote:
>
> thanks for the replies !,
> but actually I did figure out how to kill it
> but pb_cancel_backend didn't work.  here's some notes:
> this has been hung for 5 days:
> ns      |   32681 | nssql   | <IDLE> in transaction | f       | 2010-12-01
> 15

Right, pg_cancel_backend() isn't going to help when the session you're
trying to kill is '<IDLE> in transaction' -- there's no query to be
killed. If this '<IDLE> in transaction' session was causing problems
by blocking other transactions, you should look at the application
running these queries and figure out why it's hanging out in this
state. Staying like that for 5 days is not a good sign, and can cause
also problems with e.g. autovacuum.

[snip]

> but it still will not die
> the docs for pg_ctl state:
> "Use pb_ctl --help to see a list of supported signal names."
> doing so does indeed tell me the names:
> HUP INT QUIT ABRT TERM USR1 USR2
> but nothing about them whatseover :)

I agree this could be better documented. There's a brief mention at:
  http://www.postgresql.org/docs/current/static/app-postgres.html#AEN77350
  "To cancel a running query, send the SIGINT signal to the process
running that command."

though that snippet of information is out-of-place on a page about the
postmaster, and SIGINT vs. SIGTERM for individual backends isn't
discussed there at any rate.

At any rate, as you discovered, you have to send SIGTERM to the
backend to kill off an '<IDLE> in transaction' session. If you're
using 8.4 or newer, you have pg_terminate_backend() as a SQL wrapper
for SIGTERM. If you're using an older version, be careful, see e.g.
  http://archives.postgresql.org/pgsql-admin/2010-04/msg00274.php

Josh