Parallel Executors [was RE: Threaded Sorting]
От | Curtis Faith |
---|---|
Тема | Parallel Executors [was RE: Threaded Sorting] |
Дата | |
Msg-id | DMEEJMCDOJAKPPFACMPMKEDKCEAA.curtis@galtair.com обсуждение исходный текст |
Ответ на | Re: Threaded Sorting (Tom Lane <tgl@sss.pgh.pa.us>) |
Список | pgsql-hackers |
tom lane wrote: > "Curtis Faith" <curtis@galtair.com> writes: > > What about splitting out parsing, optimization and plan generation from > > execution and having a separate pool of exececutor processes. > > > As an optimizer finished with a query plan it would initiate execution > > by grabbing an executor from a pool and passing it the plan. > > So different executors would potentially handle the queries from a > single transaction? How will you deal with pushing transaction-local > state from one to the other? > > Even if you restrict it to switching at transaction boundaries, you > still have session-local state (at minimum user ID and SET settings) > to worry about. Hmmm, what transaction boundaries did you mean? Since we are talking about single statement parallization, there must be some specific internal semantics that you believe need isolation. It seems like we'd be able to get most of the benefit and restrict the parallization in a way that would preserve this isolation but I'm curious what you were specifically referring to? The current transaction/user state seems to be stored in process global space. This could be changed to be a sointer to a struct stored in a back-end specific shared memory area which would be accessed by the executor process at execution start. The backend would destroy and recreate the shared memory and restart execution in the case where an executor process dies much like the postmaster does with backends now. To the extent the executor process might make changes to the state, which I'd try to avoid if possible (don't know if it is), the executors could obtain locks, otherwise if the executions were constrained to isolated elements (changes to different indexes for example) it seems like it would be possible using an architecture where you have: Main Executor: Responsible for updating global meta data from each sub-executor and assembling the results of multiple executions. In the case of multiple executor sorts, the main executor would perform a merge sort on the results of it and it's subordinates pre-sorted sub-sets of the relation. Subordinate Executor: Executes sub-plans and returns results or meta-data update information into front-end shared memory directly. To make this optimal, the index code would have to be changed to support the idea of partial scans. In the case of btrees it would be pretty easy using the root page to figure out what index values delineated different 1/2's, 1/3's, 1/4's etc. of the index space. I'm not sure what you'd have to do to support this for table scans as I don't know the PostgreSQL tuple storage mechanism, yet. This does not seem like too much architectural complexity or performance overhead (even for the single executor case) for a big gain for complex query performance. > Being able to apply multiple CPUs to a single query is attractive, > but I've not yet seen schemes for it that don't look like the extra > CPU power would be chewed up in overhead :-(. Do you remember specifc overhead problems/issues? - Curtis
В списке pgsql-hackers по дате отправления: