Re: [v9.5] Custom Plan API

> > Let me list up the things to be clarified / developed randomly.
> >
> > * Join replacement by FDW; We still don't have consensus about join
> replacement
> >   by FDW. Probably, it will be designed to remote-join implementation
> primarily,
> >   however, things to do is similar. We may need to revisit the Hanada-san's
> >   proposition in the past.
> 
> Agreed. We need to push down joins into FDWs and we need to push down
> aggregates also, so they can be passed to FDWs. I'm planning to look at
> aggregate push down.
> 
Probably, it's a helpful feature.

> > * Lookaside for ANY relations; I want planner to try GPU-scan for any
> relations
> >   once installed, to reduce user's administration cost.
> >   It needs lookaside allow to specify a particular foreign-server, not
> foreign-
> >   table, then create ForeignScan node that is not associated with a
> particular
> >   foreign-table.
> 
> IMHO we would not want to add indexes to every column, on every table, nor
> would we wish to use lookaside for all tables. It is a good thing to be
> able to add optimizations for individual tables. GPUs are not good for
> everything; it is good to be able to leverage their strengths, yet avoid
> their weaknesses.
> 
> If do you want that, you can write an Event Trigger that automatically adds
> a lookaside for any table.
> 
It may be a solution if we try to replace scan on a relation by a ForeignScan,
in other words, a case when we can describe 1:1 relationship between a table
and a foreign-table; being alternatively scanned.

Is it possible to fit a case when a ForeignScan replaces a built-in Join plans?
I don't think it is a realistic assumption to set up lookaside configuration
for all the possible combination of joins, preliminary.

I have an idea; if lookaside accept a function, foreign-server or something
subjective entity as an alternative path, it will be able to create paths
on the fly, not only preconfigured foreign-tables.
This idea will take two forms of DDL commands as:
 CREATE LOOKASIDE <name> ON <target reltaion>   TO <alternative table/matview/foreign table/...>;
 CREATE LOOKASIDE <name> ON <target relation>   EXECUTE <path generator function>;

Things to do internally is same. TO- form kicks a built-in routine, instead
of user defined function, to add alternative scan/join paths according to
the supplied table/matview/foreign table and so on.


> > * ForeignScan node that is not associated with a particular foreign-table.
> >   Once we try to apply ForeignScan node instead of Sort or Aggregate,
> existing
> >   FDW implementation needs to be improved. These nodes scan on a
> materialized
> >   relation (generated on the fly), however, existing FDW code assumes
> >   ForeignScan node is always associated with a particular foreign-table.
> >   We need to eliminate this restriction.
> 
> I don't think we need to do that, given the above.
> 
It makes a problem if ForeignScan is chosen as alternative path of Join.

The target-list of Join node are determined according to the query form
on the fly, so we cannot expect a particular TupleDesc to be returned
preliminary. Once we try to apply ForeignScan instead of Join node, it
has to have its TupleDesc depending on a set of joined relations.

I think, it is more straightforward approach to allow ForeignScan that
is not associated to a particular (cataloged) relations.

> > * FDW method for MultiExec. In case when we can stack multiple ForeignScan
> >   nodes, it's helpful to support to exchange scanned tuples in their own
> >   data format. Let's assume two ForeignScan nodes are stacked. One
> performs
> >   like Sort, another performs like Scan. If they internally handle column-
> >   oriented data format, TupleTableSlot is not a best way for data
> exchange.
> 
> I agree TupleTableSlot may not be best way for bulk data movement. We
> probably need to look at buffering/bulk movement between executor nodes
> in general, which would be of benefit for the FDW case also.
> This would be a problem even for Custom Scans as originally presented also,
> so I don't see much change there.
> 
Yes. I is the reason why my Custom Scan proposition supports MultiExec method.

> > * Lookaside on the INSERT/UPDATE/DELETE. Probably, it can be implemented
> >   using writable FDW feature. Not a big issue, but don't forget it...
> 
> Yes, possible.
> 
> 
> I hope these ideas make sense. This is early days and there may be other
> ideas and much detail yet to come.
> 
I'd like to agree general direction. My biggest concern towards FDW is
transparency for application. If lookaside allows to redirect a reference
towards scan/join on regular relations by ForeignScan (as an alternative
method to execute), here is no strong reason to stick on custom-plan.

However, existing ForeignScan node does not support to work without
a particular foreign table. It may become a restriction if we try to
replace Join node by ForeignScan, and it is my worry.
(Even it may be solved during Join replacement by FDW works.)

One other point I noticed.

* SubPlan support; if an extension support its special logic to join relations, but don't want to support various
methodto scan relations, it is natural to leverage built-in scan logics (like SeqScan, ...). I want ForeignScan to
supportto have SubPlans if FDW driver has capability. I believe it can be implemented according to the existing manner,
butwe need to expose several static functions to handle plan-tree recursively.
 

Thanks,
--
NEC OSS Promotion Center / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>