GROUP BY / ORDER BY string is very slow

Поиск
Список
Период
Сортировка
От Oleg Broytmann
Тема GROUP BY / ORDER BY string is very slow
Дата
Msg-id Pine.SOL2.3.96.SK.990121140402.12992A-100000@sun.med.ru
обсуждение исходный текст
Список pgsql-hackers
Hello!
  I use a modified version of contrib/apache_logginig. The table I am
using:

CREATE TABLE combinedlog (  host text,  accdate abstime,  request text,  authuser text,  cookie text,  referer text,
useragenttext,  stime int2,  status int2,  bytes int4
 
);
  Once a month I run a very simple script to put WWW logs into the table.
The very loading is not fast (I am running postmaster with -F, I use
BEGIN/END and I drop indicies before loading. I remember when I started
using BEGIN/END loading speed up a bit, but not significantly), but is not
my biggest concern. What is worse is spped of my queries.  After inserting into the table, I run this shell script:

sel_f() {  field=$1  psql -d ce_wwwlog -c "SELECT COUNT($field), $field FROM raw_combinedlog GROUP BY 2 ORDER BY 1
DESC;"
}

for i in host request referer useragent; do  sel_f $i > by-$i
done
  This works very, very slow. I tried to use indicies:

CREATE INDEX host ON combinedlog (host);
CREATE INDEX request ON combinedlog (request);
CREATE INDEX referer ON combinedlog (referer);
CREATE INDEX useragent ON combinedlog (useragent);
  but indicies do not help much, and indexing time is so big, that sum of
CREATE INDEX + SELECT is even bigger :(
  Why is it so slow? How can I speed it up?
  I am running postgres compiled with --enable-locale. that is, for every
string comparision there are 2 (two) malloc calls and one strcoll. Can I
increase speed turning strcoll off? If so, postgres need a SET command to
turn localization temporary off. I can hack in, as I already rewrote
localization stuff a year ago. The only thing I want to hear from postgres
community (hackers, actually :) is how it should be named:  SET STRCOLL=off
or such?  I remember when I submitted my locale patch there was a discussion on
how to do it the Right Way, but I didn't remember the conlusion. What
finally we decided? I want to add command (if I should to) that is
compliant with other stuff here.

Oleg.
----  Oleg Broytmann     http://members.tripod.com/~phd2/     phd2@earthling.net          Programmers don't die, they
justGOSUB without RETURN.
 



В списке pgsql-hackers по дате отправления:

Предыдущее
От: "Thomas G. Lockhart"
Дата:
Сообщение: Re: [HACKERS] Re: Beta test of Postgresql 6.5
Следующее
От: Roberto Joao Lopes Garcia
Дата:
Сообщение: Performance test with -F (Was Postgres Speed or lack thereof)