Re: [v9.5] Custom Plan API

> -----Original Message-----
> From: Simon Riggs [mailto:simon@2ndQuadrant.com]
> Sent: Wednesday, May 07, 2014 5:02 PM
> To: Kaigai Kouhei(海外 浩平)
> Cc: Tom Lane; Robert Haas; Andres Freund; PgHacker; Stephen Frost; Shigeru
> Hanada; Jim Mlodgenski; Peter Eisentraut; Kohei KaiGai
> Subject: Re: [v9.5] Custom Plan API
> 
> On 7 May 2014 08:17, Kouhei Kaigai <kaigai@ak.jp.nec.com> wrote:
> 
> > Let me clarify. This mechanism allows to add alternative scan/join
> > paths including built-in ones, not only custom enhanced plan/exec node,
> isn't it?
> > Probably, it is a variation of above proposition if we install a
> > handler function that proposes built-in path nodes towards the request
> for scan/join.
> 
> Yes, I am looking for a way to give you the full extent of your requirements,
> within the Postgres framework. I have time and funding to assist you in
> achieving this in a general way that all may make use of.
> 
> > Not only alternative data structure, alternative method to scan/join
> > towards same data structure is also important, isn't it?
> 
> Agreed. My proposal is that if the planner allows the lookaside to an FDW
> then we pass the query for full execution on the FDW. That means that the
> scan, aggregate and join could take place via the FDW. i.e.
> "Custom Plan" == lookaside + FDW
> 
> Or put another way, if we add Lookaside then we can just plug in the pgstrom
> FDW directly and we're done. And everybody else's FDW will work as well,
> so Citus etcc will not need to recode.
> 
Hmm. That sounds me, you intend to make FDW perform as a central facility
to host pluggable plan/exec stuff. Even though we have several things to be
clarified, I also think it's a direction worth to investigate.

Let me list up the things to be clarified / developed randomly.

* Join replacement by FDW; We still don't have consensus about join replacement by FDW. Probably, it will be designed
toremote-join implementation primarily, however, things to do is similar. We may need to revisit the Hanada-san's
propositionin the past.
 

* Lookaside for ANY relations; I want planner to try GPU-scan for any relations once installed, to reduce user's
administrationcost. It needs lookaside allow to specify a particular foreign-server, not foreign- table, then create
ForeignScannode that is not associated with a particular foreign-table.
 

* ForeignScan node that is not associated with a particular foreign-table. Once we try to apply ForeignScan node
insteadof Sort or Aggregate, existing FDW implementation needs to be improved. These nodes scan on a materialized
relation(generated on the fly), however, existing FDW code assumes ForeignScan node is always associated with a
particularforeign-table. We need to eliminate this restriction.
 

* FDW method for MultiExec. In case when we can stack multiple ForeignScan nodes, it's helpful to support to exchange
scannedtuples in their own data format. Let's assume two ForeignScan nodes are stacked. One performs like Sort, another
performslike Scan. If they internally handle column- oriented data format, TupleTableSlot is not a best way for data
exchange.

* Lookaside on the INSERT/UPDATE/DELETE. Probably, it can be implemented using writable FDW feature. Not a big issue,
butdon't forget it...
 

How about your opinion?

Thanks,
--
NEC OSS Promotion Center / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>