Re: Support Parallel Query Execution in Executor

"Tom Lane" <tgl@sss.pgh.pa.us> wrote
>
> This is exactly the bit of optimism I was questioning.  We've already
> been sweating blood trying to reduce multiprocessor contention on data
> structures in which collisions ought to be avoidable (ie, buffer arrays
> where you hope not everyone is hitting the same buffer at once).  I
> think passing large volumes of data between different processes is going
> to incur quite a lot of locking overhead, pipeline stalls for cache line
> transfers, etc, etc, because heavy contention for the transfer buffer is
> simply not going to be avoidable.
>

Yes, there is overhead. Let's say how can we tackle them one by one:

* lock contention of transfer buffer *
A general way is partitioning. We can try to alleviate the lock contention 
by sharing transfer buffer only between each two processes (master and 
slave).

* large amount of data *
Thinking of the simplest master-slave seqscan senario, I am copying the 
HeapTuple so far (like the code in sort read/write tuple), but potentially 
we can only pass the offset and pinned page, which will require the slave 
process pinned the page for a little longer time, say every 5 pages. I 
didn't see any application can cause Postgres pinned a lot pages, so this 
might be one step to reduce the data volumn.

On the other hand, why do we need parallel execution anyway? Because if we 
got a 4 way SMP with strong IO device, we may want to trade the performance 
with more resources for some queries.


Regards,
Qingqing