Re: ETL - sql orchestrator is stuck when there is not sleep() between queries

Поиск
Список
Период
Сортировка
От Justin Pryzby
Тема Re: ETL - sql orchestrator is stuck when there is not sleep() between queries
Дата
Msg-id 20210708123320.GB22043@telsasoft.com
обсуждение исходный текст
Ответ на ETL - sql orchestrator is stuck when there is not sleep() between queries  (Allan Barrielle <allan.barrielle@gmail.com>)
Ответы Re: ETL - sql orchestrator is stuck when there is not sleep() between queries
Список pgsql-performance
On Thu, Jul 08, 2021 at 01:00:28AM +0200, Allan Barrielle wrote:
> All is fine, and can work great.
> But sometimes, some queries that used to take about 20 secs to complete can
> suddenly end in 5mins.
> Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit
> of transform) FROM TABLE). No update, nothing, it’s dead simple.
> 
> Nothing works, it’s very random, some query won’t simply work ( even after
> hours ).

When it doesn't work, you could check SELECT * FROM pg_stat_activity, and
SELECT pg_blocking_pids(pid), * FROM pg_locks, to see what's going on.

> Important all queries have the same shape -> CREATE TABLE SELECT AS *(a bit
> of transform) FROM TABLE). No update, nothing, it’s dead simple.
> We are just trying to copy a table from schema1, to schema2, to schema3 and
> finally schema3. That’s it.

Is it true that the SELECTs have no joins in them ?

Did this ever work better or differently under different versions of postgres ?

> Why does the query never end even after hours ? Why there is no log about
> where the query is stuck.

Please send your nondefault config.
https://wiki.postgresql.org/wiki/Server_Configuration

Also enable logging (I just added this to the wiki).
https://wiki.postgresql.org/wiki/Slow_Query_Questions#Enable_Logging

It'd be very useful to get "explain analyze" for a working query and for a
stuck query.  It sound like the stuck query never finishes, so maybe the second
part is impossible (?)

But it'd be good to get at least "explain" output.  You'd have to edit your sql
script to run an "explain" before each query, and run it, logging the ouput,
until you capture the plan for a stuck query.  Save the output and send here,
along with the query plan for a working query.

-- 
Justin



В списке pgsql-performance по дате отправления:

Предыдущее
От: Joel Frid
Дата:
Сообщение: Strange execution plan
Следующее
От: Imre Samu
Дата:
Сообщение: Re: ETL - sql orchestrator is stuck when there is not sleep() between queries