On Sat, Mar 1, 2014 at 8:49 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> This critique seems pretty odd to me. I haven't had the time to look
>> at this patch set, but I don't see why anyone would want to use the
>> background worker facility for GPU acceleration, which is what
>> KaiGai's trying to accomplish here.
>
> Eh, that didn't come out quite right. I had intended it to be more
> along the lines of "look at what Robert's doing".
>
> I was trying to point out that parallel query execution is coming soon
> thanks to the work on background worker and that parallel query
> execution might drive changes to the way nodes interact in the executor
> driving a need to change the API. In other words, CustomScan could
> easily end up being broken by that change and I'd rather we not have to
> worry about such breakage.
I think the relation is pretty tangential. I'm worried about the
possibility that the Custom Scan API is broken *ab initio*, but I'm
not worried about a conflict with parallel query.
>> I seriously doubt there's any real conflict with parallelism here.
>> Parallelism may indeed add more ways to scan a relation
>> (ParallelSeqScan, ParallelIndexScan?) but that doesn't mean that we
>> shouldn't have a Custom Scan node too.
>
> What about parallel execution through the tree itself, rather than just
> at specific end nodes like SeqScan and IndexScan? Or parallel
> aggregates? I agree that simple parallel SeqScan/IndexScan isn't going
> to change any of the interfaces, but surely we're going for more than
> that. Indeed, I'm wishing that I had found more time to spend on just
> a simple select-based Append node which could parallelize I/O across
> tablespaces and FDWs underneath the Append.
Well, as I said in another recent reply that you probably got after
sending this, when you split between nodes, that mostly just has to do
with how you funnel the tuples from one node to another. The nodes
themselves probably don't even need to know. Or at least that's what
I'd hope.
I don't see that parallelizing Append is any easier than any other
problem in this space. There's no parallel I/O facility, so you need
a background worker per append branch to wait on I/O. And you have
all the problems of making sure that the workers have the same
snapshot, making sure they don't self-deadlock, etc. that you have for
any other case.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company