Re: [DESIGN] ParallelAppend

I have to say that I really like the thought of us having parallel enabled stuff in Postgres, but I also have to say that I don't think inventing all these special parallel node types is a good idea. If we think about everything that we can parallelise...

Perhaps.... sort, hash join, seqscan, hash, bitmap heap scan, nested loop. I don't want to debate that, but perhaps there's more, perhaps less.

Are we really going to duplicate all of the code and add in the parallel stuff as new node types?

My other concern here is that I seldom hear people talk about the planner's architectural lack of ability to make a good choice about how many parallel workers to choose. Surely to properly calculate costs you need to know the exact number of parallel workers that will be available at execution time, but you need to know this at planning time!? I can't see how this works, apart from just being very conservative about parallel workers, which I think is really bad, as many databases have busy times in the day, and also quiet times, generally quiet time is when large batch stuff gets done, and that's the time that parallel stuff is likely most useful. Remember queries are not always planned just before they're executed. We could have a PREPAREd query, or we could have better plan caching in the future, or if we build some intelligence into the planner to choose a good number of workers based on the current server load, then what's to say that the server will be under this load at exec time? If we plan during a quiet time, and exec in a busy time all hell may break loose.

I really do think that existing nodes should just be initialized in a parallel mode, and each node type can have a function to state if it supports parallelism or not. 

I'd really like to hear more opinions in the ideas I discussed here: 

http://www.postgresql.org/message-id/CAApHDvp2STf0=pQfpq+e7WA4QdYmpFM5qu_YtUpE7R0jLnH82Q@mail.gmail.com 

This design makes use of the Funnel node that Amit has already made and allows more than 1 node to be executed in parallel at once.

It appears that parallel enabling the executor node by node is fundamentally locked into just 1 node being executed in parallel, then perhaps a Funnel node gathering up the parallel worker buffers and streaming those back in serial mode. I believe by design, this does not permit a whole plan branch from executing in parallel and I really feel like doing things this way is going to be very hard to undo and improve later. I might be too stupid to figure it out, but how would parallel hash join work if it can't gather tuples from the inner and outer nodes in parallel?

Sorry for the rant, but I just feel like we're painting ourselves into a corner by parallel enabling the executor node by node.

Apologies if I've completely misunderstood things.

Regards 

David Rowley