Re: On partitioning

On 18-12-2014 AM 04:52, Robert Haas wrote:
> On Wed, Dec 17, 2014 at 1:53 PM, Josh Berkus <josh@agliodbs.com> wrote:
>>
>> Sure.  But there's a big difference between "we're going to take these
>> steps and that problem will be fixable eventually" and "we're going to
>> retain features of the current partitioning system which make that
>> problem impossible to fix."  The drift of discussion on this thread
>> *sounded* like the latter, and I've been calling attention to the issue
>> in an effort to make sure that it's not.
>>
>> Last week, I wrote a longish email listing out the common problems users
>> have with our current partitioning as a way of benchmarking the plan for
>> new partitioning.  While some people responded to that post, absolutely
>> nobody discussed the list of issues I gave.  Is that because there's
>> universal agreement that I got the major issues right?  Seems doubtful.
> 
> I agreed with many of the things you listed but not all of them.
> However, I don't think it's realistic to burden whatever patch Amit
> writes with the duty of, for example, making global indexes work.
> That's a huge problem all of its own.  Now, conceivably, we could try
> to solve that as part of the next patch by insisting that the
> "partitions" have to really be block number ranges within a single
> relfilenode rather than separate relfilenodes as they are today ...
> but I think that's a bad design which we would likely regret bitterly.
> I also think that it would likely make what's being talked about here
> so complicated that it will never go anywhere.  I think it's better
> that we focus on solving one problem really well - storing metadata
> for partition boundaries in the catalog so that we can do efficient
> tuple routing and partition pruning - and leave the other problems for
> later.
> 

Yes, I think partitioning as a whole is a BIG enough project that we
need to tackle it as a series of steps each of which is a discussion of
its own. The first step might as well be discussing how we represent a
partitioned table. We have a number of design decisions to make during
this step itself and we would definitely want to reach a consensus on
these points.

Things like where we indicate if a table is partitioned (pg_class), what
the partition key looks like, where it is stored, what the partition
definition looks like, where it is stored, how we represent arbitrary
number of levels in partitioning hierarchy, how we implement that only
leaf level relations in a hierarchy have storage, what are implications
of all these choices, etc. Some of these points are being discussed.

I agree that while we are discussing these points, we could also be
discussing how we solve problems of existing partitioning implementation
using whatever the above things end up being. Proposed approaches to
solve those problems might be useful to drive the first step as well or
perhaps that's how it should be done anyway.

Thanks,
Amit