On Mon, Aug 31, 2015 at 11:08 AM, Bruce Momjian <bruce@momjian.us> wrote:
>
> On Mon, Aug 31, 2015 at 09:53:57AM +0900, Michael Paquier wrote:
> > Well, I have had many such discussions with XC/XL folks, and that was my
> > opinion. I have seen almost no public discussion about this because the
> > idea had almost no chance of success. If it was possible, someone would
> > have already suggested it on this list.
> >
> >
> > Or perhaps people invested in this area had other obligations or lacked
> > motivation and/or time to work to push up for things in core. That's not
> > possible to know, and what is done is done.
>
> Well, I have talked to everyone privately about this, and concluded that
> while horizontal scalability/sharding is useful, it is unlikely that the
> code volume of something like XC or XL would be accepted into the
> community, and frankly, now that we have FDWs, it is hard to imagine why
> we would _not_ go in the FDW direction.
If I recall correctly in terms of numbers, that's indeed 40k of code,
the main areas of XC code being the GTM, the planner changes for
expression and join push down, and the connection pooler for parallel
query execution.
ISTM that FDW is a portion of the puzzle, there are other pieces that
could be used toward an in-core integration, like the parallel stuff
Amit Kapila is working on to allow remote query execution in parallel
of local scans. Also, XC/XL were performing well on OLTP thanks to the
connection pooler: this should indeed be part of the FDW portion
managing the foreign scans. This may sound like a minor issue compared
to the others, but already established connections help a lot when
scaling out with foreign servers.
> Of course, people have concerns, and FDWs might need to be improved, but
> it is something worth researching. We might find out FDWs can't be used
> at all, and that we have to either add much more code to Postgres to do
> sharding, do something like pg_shard, or not implement built-in sharding
> at all, but at least it is time to research this.
I am really looking forward to hearing the arguments of the authors of
pg_shard on the matter.
> > OK, I will send you a separate email and you can then supply their email
> > addresses.
> >
> >
> > FWIW, I would be interested in that as well. I worked in this area of things
> > for a couple of years as well FWIW.
>
> OK, I will send you an email.
Thanks.
--
Michael