Re: BUG #18909: Query creates millions of temporary files and stalls

And this is explain analyze run that finishes in psql, with the default settings (i.e. with parallelism) :

                                                                                      QUERY PLAN

-----------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------
 Gather  (cost=14443102.62..111487693.90 rows=65470868 width=100) (actual time=212912.377..334892.676 rows=65514296
loops=1)
   Workers Planned: 4
   Workers Launched: 4
   ->  Nested Loop Left Join  (cost=14442102.62..104939607.10 rows=16367717 width=100) (actual
time=209806.019..317684.146rows=13102859 loops=5)
 
         ->  Parallel Hash Left Join  (cost=14442102.04..22124798.60 rows=16367717 width=60) (actual
time=209805.943..273008.489rows=13102859 loops=5)
 
               Hash Cond: (d.objid = gaps1.original_ext_source_id)
               ->  Parallel Seq Scan on disk_sample1 d  (cost=0.00..1218371.17 rows=16367717 width=60) (actual
time=37.353..25185.340rows=13095751 loops=5)
 
               ->  Parallel Hash  (cost=10307380.24..10307380.24 rows=237862624 width=16) (actual
time=169633.067..169633.068rows=190290095 loops=5)
 
                     Buckets: 67108864  Batches: 32  Memory Usage: 1919904kB
                     ->  Parallel Seq Scan on panstarrs1bestneighbour gaps1  (cost=0.00..10307380.24 rows=237862624
width=16)(actual time=132.295..117548.803
 
rows=190290095 loops=5)
         ->  Index Scan using gaia_sourcex_source_id_idx on gaia_source g  (cost=0.58..5.05 rows=1 width=48) (actual
time=0.003..0.003rows=0 loops=65514296)
 
               Index Cond: (source_id = gaps1.source_id)
 Planning Time: 1.266 ms
 JIT:
   Functions: 75
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 11.796 ms, Inlining 293.374 ms, Optimization 81.354 ms, Emission 173.338 ms, Total 559.861 ms
 Execution Time: 337814.695 ms
(18 rows)

And I did verify again that when I query through python (i.e. using the cursor) with
max_parallel_workers_per_gather=0 it finishes fine.

Also I clearly see that when I query through the cursor in python with the default settings (and when I see the issue),
itclearly uses
 
 a different plan, as opposed to just running the query in psql.
Because when running in psql I see these kinds of files in tmp
pgsql_tmp75270.0.fileset
as opposed to :
pgsql_tmp73459.0 ...

I don't think I know how to see the plan of the declare cursor query.

So summarizing:

the query produces millions of files in

1) query through cursor with default settings (max_parallel_workers_per_gather=4)
2) query through psql with no parallelism(max_parallel_workers_per_gather=0)

it works
3) query through cursor with no parallelism (max_parallel_workers_per_gather=0)
4) query through psql with default settings (max_parallel_workers_per_gather=4)

I hope it makes sense to someone &  helps.

        Sergey




The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e
buidheanncarthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.