Re: Custom Scan APIs (Re: Custom Plan node)

2014-03-01 0:36 GMT+09:00 Stephen Frost <sfrost@snowman.net>:
> * Kouhei Kaigai (kaigai@ak.jp.nec.com) wrote:
>> * Stephen Frost (sfrost@snowman.net) wrote:
>> > I don't see how you can be when there hasn't been any discussion that I've
>> > seen about how parallel query execution is going to change things for us.
>> >
>> If parallel query execution changes whole of the structure of plan nodes,
>> it will also affect to the interface of custom-scan because it is a thin-
>> abstraction of plan-node. However, if parallel execution feature is
>> implemented as one of new plan node in addition to existing one, I cannot
>> imagine a scenario that affects to the structure of another plan node.
>
> Let's just say that I have doubts that we'll be able to implement
> parallel execution *without* changing the plan node interface in some
> way which will require, hopefully minor, changes to all of the nodes.
> The issue is that even a minor change would break the custom-scan API
> and we'd immediately be in the boat of dealing with complaints regarding
> backwards compatibility.  Perhaps we can hand-wave that, and we've had
> some success changing hook APIs between major releases, but such changes
> may also be in ways which wouldn't break in obvious ways or even
> possibly be changes which have to be introduced into back-branches.
> Parallel query is going to be brand-new real soon and it's reasonable to
> think we'll need to make bug-fix changes to it after it's out which
> might even involve changes to the API which is developed for it.
>
Even if we will change the plan-node interface in the future release,
it shall not be a change that makes the existing stuff impossible.
The custom-scan API is designed to provide alternative way to scan
or join relations, in addition to the existing logic like SeqScan or
NestLoop. If this change breaks plan-node interfaces and it couldn't
implement existing stuff, it is problematic for all the stuff, not only
custom-scan node. I don't think such a change that makes impossible
to implement existing logic will be merged. Likely, the new parallel
execution feature can work together existing sequential logic and
custom-scan interface.

BTW, this kind of discussion looks like a talk with a ghost because
we cannot see the new interface according to the parallel execution
right now, so we cannot have tangible investigation whether it becomes
really serious backward incompatibility, or not.
My bet is minor one. I cannot imagine plan-node interface that does
not support existing non-parallel SeqScan or NestLoop and so on.

>> > The issue here is that we're going to be expected to maintain an interface
>> > once we provide it and so that isn't something we should be doing lightly.
>> > Particularly when it's as involved as this kind of change is with what's
>> > going on in the backend where we are nearly 100% sure to be changing things
>> > in the next release or two.
>> >
>> FDW APIs are also revised several times in the recent releases. If we can
>> design "perfect" interface from the beginning, it's best but usually hard
>> to design.
>
> Sure, but FDWs also have a *much* broader set of use-cases, in my view,
> which is also why I was pushing to work on join-push-down to happen
> there instead of having this kind of a hook interface, which I don't
> think we'd want to directly expose as part of the 'official FDW API' as
> it ends up putting all the work on the FDW with little aide, making it
> terribly likely to end up with a bunch of duplciated code in the FDWs
> from the backend to deal with it, particularly for individuals writing
> FDWs who aren't familiar with what PG already has.
>
It might not be a good idea to use postgres_fdw as a basis of proof-
of-concept to demonstrate that custom-scan can effectively host
an alternative way to join; that fetches the result set of remote-join
as if relation scan, even though it demonstrated it is possible.
So, I never mind the part-3 portion (remote join of postgres_fdw on
top of custom-scan) being dropped from the submission.

Thanks,
-- 
KaiGai Kohei <kaigai@kaigai.gr.jp>