Re: Custom Scan APIs (Re: Custom Plan node)

Поиск
Список
Период
Сортировка
От Stephen Frost
Тема Re: Custom Scan APIs (Re: Custom Plan node)
Дата
Msg-id 20140303154346.GN12995@tamriel.snowman.net
обсуждение исходный текст
Ответ на Re: Custom Scan APIs (Re: Custom Plan node)  (Robert Haas <robertmhaas@gmail.com>)
Ответы Re: Custom Scan APIs (Re: Custom Plan node)  (Robert Haas <robertmhaas@gmail.com>)
Re: Custom Scan APIs (Re: Custom Plan node)  (Ashutosh Bapat <ashutosh.bapat@enterprisedb.com>)
Список pgsql-hackers
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Sat, Mar 1, 2014 at 9:04 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > Erm, my thought was to use a select() loop which sends out I/O requests
> > and then loops around waiting to see who finishes it.  It doesn't
> > parallelize the CPU cost of getting the rows back to the caller, but
> > it'd at least parallelize the I/O, and if what's underneath is actually
> > a remote FDW running a complex query (because the other side is actually
> > a view), it would be a massive win to have all the remote FDWs executing
> > concurrently instead of serially as we have today.
>
> I can't really make sense of this.

Sorry, that was a bit hand-wavey since I had posted about it previously
here:
http://www.postgresql.org/message-id/20131104032604.GB2706@tamriel.snowman.net

It'd clearly be more involved than "just build a select() loop" and
would require adding an async mechanism.  I had been thinking about this
primairly with the idea of FDWs and you're right that it'd require more
thought to deal with getting data into/through shared_buffers.  Still,
we seqscan into a ring buffer, I'd think we could make it work but it
would require additional work.

> For FDWs, one idea might be to kick off the remote query at
> ExecInitNode() time rather than ExecProcNode() time, at least if the
> remote query doesn't depend on parameters that aren't available until
> run time.

Right, I had speculated about that also (option #2 in my earlier email).

> That actually would allow multiple remote queries to run
> simultaneously or in parallel with local work.  It would also run them
> in cases where the relevant plan node is never executed, which would
> be bad but perhaps rare enough not to worry about.

This was my primary concern, along with the fact that we explicitly says
"don't do that" in the docs for the FDW API.

> Or we could add a
> new API like ExecPrefetchNode() that tells nodes to prepare to have
> tuples pulled, and they can do things like kick off asynchronous
> queries.  But I still don't see any clean way for the Append node to
> find out which one is ready to return results first.

Yeah, that's tricky.
Thanks,
    Stephen

В списке pgsql-hackers по дате отправления:

Предыдущее
От: Euler Taveira
Дата:
Сообщение: Re: pg_dump reporing version of server & pg_dump as comments in the output
Следующее
От: Simon Riggs
Дата:
Сообщение: Re: ALTER TABLE lock strength reduction patch is unsafe