Re: Slow concurrent processing

Поиск
Список
Период
Сортировка
От Misa Simic
Тема Re: Slow concurrent processing
Дата
Msg-id CAH3i69=zLg69a+1aDzWEx3LgdoeHi8CEq_FwKKeAOG3eiH5mng@mail.gmail.com
обсуждение исходный текст
Ответ на Re: Slow concurrent processing  (Steve Crawford <scrawford@pinpointresearch.com>)
Ответы Re: Slow concurrent processing
Список pgsql-performance
Thanks Steve

Well, the full story is too complex - but point was - whatever blackbox does - it last 0.5 to 2secs per 1 processed record (maybe I was wrong but I thought the reason why it takes the time how much it needs to actually do the task -CPU/IO/memory whatever is not that important....) - so I really don't see difference between: call web service, insert row in the table (takes 3 secs) and sleep 3 seconds - insert result in the table...

if we do above task for two things sequential - it will last 6 secs...but if we do it "concurentelly" - it should last 3 secs... (in theory :) )

I was guessed somewhere is lock - but wasn't clear where/why when there are no updates - just inserts...

But I haven't know that during INSERT is done row lock on refferenced tables as well - from FK columns...

So I guess now it is cause of the problem...

We will see how it goes with insert into unlogged tables with no FK...

Many thanks,

Misa


2013/3/12 Steve Crawford <scrawford@pinpointresearch.com>
On 03/11/2013 08:55 PM, Misa Simic wrote:
Hi all,

We have one table with list of "records for processing"...

We loop trough that table and call one long runing function:

 do_the_math_for_record(record_id)...<snip>...


but - if replace do_the_math_and_save_results with pg_sleep(1); To simulate long running function so processing each row - last 1 sec...

Sequential processing last as expected 300 seconds!

Concurrent processing last faster with higher number of concurrent connections - about 30 seconds with 30 connections! (much faster - and expected...)

however, if we return our: do_the_math_and_save_results - we can't get better results in concurrent processing...

Sleep will not have any significant impact on CPU, memory or disk use and thus is not a simulation of actual processing.

All you have really shown us so far is a black box. Please provide an overview of your schemas and the type of processing you are attempting on them.

Cheers,
Steve


В списке pgsql-performance по дате отправления:

Предыдущее
От: Steve Crawford
Дата:
Сообщение: Re: Slow concurrent processing
Следующее
От: Steve Crawford
Дата:
Сообщение: Re: Slow concurrent processing