Massive table (500M rows) update nightmare

Поиск
Список
Период
Сортировка
Искать
От
Carlo Stonebanks
Тема
Massive table (500M rows) update nightmare
Дата
Msg-id
hi41os$eiv$1@news.hub.org
Список
Дерево обсуждения
Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Eduardo Morras <emorras@s21sec.com>
Re: Massive table (500M rows) update nightmare Leo Mannhart <leo.mannhart@beecom.ch>
Re: Massive table (500M rows) update nightmare "Kevin Grittner" <Kevin.Grittner@wicourts.gov>
Re: Massive table (500M rows) update nightmare Leo Mannhart <leo.mannhart@beecom.ch>
Re: Massive table (500M rows) update nightmare Ludwik Dylag <ldylag@gmail.com>
Re: Massive table (500M rows) update nightmare "Greg Sabino Mullane" <greg@turnstep.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare marcin mank <marcin.mank@gmail.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare "Kevin Grittner" <Kevin.Grittner@wicourts.gov>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare "Kevin Grittner" <Kevin.Grittner@wicourts.gov>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Kenneth Marshall <ktm@rice.edu>
Re: Massive table (500M rows) update nightmare "Kevin Grittner" <Kevin.Grittner@wicourts.gov>
Re: Massive table (500M rows) update nightmare Greg Smith <greg@2ndquadrant.com>
Re: Massive table (500M rows) update nightmare Scott Marlowe <scott.marlowe@gmail.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Scott Marlowe <scott.marlowe@gmail.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Andreas Kretschmer <akretschmer@spamfence.net>
Re: Massive table (500M rows) update nightmare Pierre Frédéric Caillaud<lists@peufeu.com>
New server to improve performance on our large and busy DB - advice? "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: New server to improve performance on our large and busy DB - advice? Greg Smith <greg@2ndquadrant.com>
Re: New server to improve performance on our large and busy DB - advice? "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: New server to improve performance on our large and busy DB - advice? "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: New server to improve performance on our large and busy DB - advice? Greg Smith <greg@2ndquadrant.com>
Re: New server to improve performance on our large and busy DB -advice? "Joshua D. Drake" <jd@commandprompt.com>
Re: New server to improve performance on our large and busy DB - advice? marcin mank <marcin.mank@gmail.com>
Re: New server to improve performance on our large and busy DB - advice? Scott Marlowe <scott.marlowe@gmail.com>
Re: New server to improve performance on our large and busy DB - advice? Alvaro Herrera <alvherre@commandprompt.com>
Re: New server to improve performance on our large and busy DB - advice? Scott Marlowe <scott.marlowe@gmail.com>
Re: New server to improve performance on our large and busy DB - advice? Alvaro Herrera <alvherre@commandprompt.com>
Re: New server to improve performance on our large and busy DB - advice? Devrim GÜNDÜZ <devrim@gunduz.org>
Re: New server to improve performance on our large and busy DB - advice? Scott Marlowe <scott.marlowe@gmail.com>
Re: New server to improve performance on our large and busy DB - advice? "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: New server to improve performance on our large and busy DB - advice? Scott Marlowe <scott.marlowe@gmail.com>
Re: New server to improve performance on our large and busy DB - advice? "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: New server to improve performance on our large and busy DB - advice? "Kevin Grittner" <Kevin.Grittner@wicourts.gov>
Re: New server to improve performance on our large and busy DB - advice? Robert Haas <robertmhaas@gmail.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Craig James <craig_james@emolecules.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Re: Massive table (500M rows) update nightmare Scott Marlowe <scott.marlowe@gmail.com>
Re: Massive table (500M rows) update nightmare "Carlo Stonebanks" <stonec.register@sympatico.ca>
Our DB has an audit table which is 500M rows and growing. (FYI the objects 
being audited are grouped semantically, not individual field values).

Recently we wanted to add a new feature and we altered the table to add a 
new column. We are backfilling this varchar(255) column by writing a TCL 
script to page through the rows (where every update is a UPDATE ... WHERE id 
 >= x AND id < x+10 and a commit is performed after every 1000 updates 
statement, i.e. every 10000 rows.)

We have 10 columns, six of which are indexed. Rough calculations suggest 
that this will take two to three weeks to complete on an 8-core CPU with 
more than enough memory.

As a ballpark estimate - is this sort of performance for an 500M updates 
what one would expect of PG given the table structure (detailed below) or 
should I dig deeper to look for performance issues?

As always, thanks!

Carlo

Table/index structure:

CREATE TABLE mdx_core.audit_impt
(
  audit_impt_id serial NOT NULL,
  impt_session integer,
  impt_version character varying(255),
  impt_name character varying(255),
  impt_id integer,
  target_table character varying(255),
  target_id integer,
  target_op character varying(10),
  note text,
  source_table character varying(255),
  CONSTRAINT audit_impt_pkey PRIMARY KEY (audit_impt_id)
)

CREATE INDEX audit_impt_impt_id_idx
  ON mdx_core.audit_impt
  USING btree
  (impt_id);
CREATE INDEX audit_impt_impt_name
  ON mdx_core.audit_impt
  USING btree
  (impt_name, impt_version);
CREATE INDEX audit_impt_session_idx
  ON mdx_core.audit_impt
  USING btree
  (impt_session);
CREATE INDEX audit_impt_source_table
  ON mdx_core.audit_impt
  USING btree
  (source_table);
CREATE INDEX audit_impt_target_id_idx
  ON mdx_core.audit_impt
  USING btree
  (target_id, audit_impt_id);
CREATE INDEX audit_impt_target_table_idx
  ON mdx_core.audit_impt
  USING btree
  (target_table, target_id, audit_impt_id);



В списке pgsql-performance по дате отправления
От: Craig Ringer
Дата:
От: Michael Ruf
Дата:
FAQ