Обсуждение: [HACKERS] Optimise default partition scanning while adding new partition

Hi Jeevan,

On 2017/09/12 18:22, Jeevan Ladhe wrote:
> Commit 6f6b99d1335be8ea1b74581fc489a97b109dd08a introduced default
> partitioning support. This commit added a new function
> check_default_allows_bound(),
> which checks if there exists a row in the default partition that would
> belong to
> the new partition being added. If it finds one, it throws an error. Before
> taking
> the decision to scan the default partition, this function checks if there
> are
> existing constraints on default partition that would imply the new partition
> constraints, if yes it skips scanning the default partition, otherwise it
> scans the
> default partition and its children(if any). But, while doing so the current
> code
> misses the fact that there can be constraints on the child of default
> partition
> such that they would imply the constraints of the new partition being added,
> and hence individual child scan can also be skipped.
> Attached is the patch which does this.
> 
> This is previously discussed in default partitioning thread[1], and decision
> was made that we can take this a separate patch rather than as a part of the
> default partitioning support.

Patch looks fine to me.  By the way, why don't we just say "Can we skip
scanning part_rel?" in the comment before the newly added call to
PartConstraintImpliedByRelConstraint()?  We don't need to repeat the
explanation of what it does at the every place we call it.

> Amit Langote has a similar patch[2] for scanning the children of a
> partitioned
> table which is being attached as partition of another partitioned table.

I just posted my rebased patch there [1].

Thanks,
Amit

[1]
https://www.postgresql.org/message-id/a83a0899-19f5-594c-9aac-3ba0f16989a1%40lab.ntt.co.jp



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Sep 14, 2017 at 4:03 AM, Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
> Thanks Amit for reviewing.
>> Patch looks fine to me.  By the way, why don't we just say "Can we skip
>> scanning part_rel?" in the comment before the newly added call to
>> PartConstraintImpliedByRelConstraint()?  We don't need to repeat the
>> explanation of what it does at the every place we call it.
>
> I have changed the comment.
> PFA.

I'm probably missing something stupid, but why does this happen?
ALTER TABLE list_parted2 ATTACH PARTITION part_7 FOR VALUES IN (7);
-INFO:  partition constraint for table "list_parted2_def" is implied
by existing constraintsERROR:  partition constraint is violated by some row

Based on the regression test changes made up higher in the file, I'd
expect that line to be replaced by two lines, one for
list_parted2_def_p1 and another for list_parted2_def_p2, because at
this point, list_parted2_def is a partitioned table with those two
children, and they seem to have appropriate constraints.

list_parted2_def_p1 hasCheck constraints:    "check_a" CHECK (a = ANY (ARRAY[21, 22]))

list_parted2_def_p2 hasCheck constraints:    "check_a" CHECK (a = ANY (ARRAY[31, 32]))

Well, if a is 21 or 22 for the first, or 31 or 32 for the second, then
it's not 7.  Or so I would think.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/09/15 0:59, Robert Haas wrote:
> On Thu, Sep 14, 2017 at 4:03 AM, Jeevan Ladhe
> <jeevan.ladhe@enterprisedb.com> wrote:
>> Thanks Amit for reviewing.
>>> Patch looks fine to me.  By the way, why don't we just say "Can we skip
>>> scanning part_rel?" in the comment before the newly added call to
>>> PartConstraintImpliedByRelConstraint()?  We don't need to repeat the
>>> explanation of what it does at the every place we call it.
>>
>> I have changed the comment.
>> PFA.
> 
> I'm probably missing something stupid, but why does this happen?
> 
>  ALTER TABLE list_parted2 ATTACH PARTITION part_7 FOR VALUES IN (7);
> -INFO:  partition constraint for table "list_parted2_def" is implied
> by existing constraints
>  ERROR:  partition constraint is violated by some row
> 
> Based on the regression test changes made up higher in the file, I'd
> expect that line to be replaced by two lines, one for
> list_parted2_def_p1 and another for list_parted2_def_p2, because at
> this point, list_parted2_def is a partitioned table with those two
> children, and they seem to have appropriate constraints.
> 
> list_parted2_def_p1 has
>  Check constraints:
>      "check_a" CHECK (a = ANY (ARRAY[21, 22]))
> 
> list_parted2_def_p2 has
>  Check constraints:
>      "check_a" CHECK (a = ANY (ARRAY[31, 32]))
> 
> Well, if a is 21 or 22 for the first, or 31 or 32 for the second, then
> it's not 7.  Or so I would think.

ISTM, Jeevan's patch modifies check_default_allows_bound(), which only
DefineRelation() calls as things stand today.  IOW, it doesn't handle the
ATTACH PARTITION case, so you're not seeing the lines for
list_parted2_def_p1 and list_parted2_def_p2, because ATExecAttachPartition
doesn't yet know how to skip the validation scan for default partition's
children.

I wonder if we should call check_default_allows_bound() from
ATExecAttachPartition(), too, instead of validating updated default
partition constraint using ValidatePartitionConstraints()?  That is, call
the latter only to validate the partition constraint of the table being
attached and call check_default_allows_bound() to validate the updated
default partition constraint.  That way, INFO/ERROR messages related to
default partition constraint are consistent across the board.  I hacked
that up in the attached 0001.  The patch also modifies the INFO message
that check_default_allows_bound() emits when the scan is skipped to make
it apparent that a default partition is involved.  (Sorry, this should've
been a review comment I should've posted before the default partition
patch was committed.)

Then apply 0002, which is Jeevan's patch.  With 0001 in place, Robert's
complaint is taken care of.

Then comes 0003, which is my patch to teach ValidatePartitionConstraints()
to skip child table scans using their existing constraint.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Sep 15, 2017 at 2:00 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I wonder if we should call check_default_allows_bound() from
> ATExecAttachPartition(), too, instead of validating updated default
> partition constraint using ValidatePartitionConstraints()?  That is, call
> the latter only to validate the partition constraint of the table being
> attached and call check_default_allows_bound() to validate the updated
> default partition constraint.  That way, INFO/ERROR messages related to
> default partition constraint are consistent across the board.

I believe the intended advantage of the current system is that if you
specify multiple operations in a single ALTER TABLE command, you only
do one scan rather than having a second scan per operation.  If that's
currently working, we probably don't want to make it stop working.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sat, Sep 16, 2017 at 12:59 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Sep 15, 2017 at 2:00 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I wonder if we should call check_default_allows_bound() from
>> ATExecAttachPartition(), too, instead of validating updated default
>> partition constraint using ValidatePartitionConstraints()?  That is, call
>> the latter only to validate the partition constraint of the table being
>> attached and call check_default_allows_bound() to validate the updated
>> default partition constraint.  That way, INFO/ERROR messages related to
>> default partition constraint are consistent across the board.
>
> I believe the intended advantage of the current system is that if you
> specify multiple operations in a single ALTER TABLE command, you only
> do one scan rather than having a second scan per operation.  If that's
> currently working, we probably don't want to make it stop working.

OK.

How about squash Jeevan's and my patch, so both
check_default_allows_bound() and ValidatePartitionConstraints() know
to scan default partition's children and there won't be any surprises
in the regression test output as you found after applying just the
Jeevan's patch.  Unfortunately, I'm not able to post such a patch
right now.

Thanks,
Amit


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/09/16 1:57, Amit Langote wrote:
> On Sat, Sep 16, 2017 at 12:59 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I believe the intended advantage of the current system is that if you
>> specify multiple operations in a single ALTER TABLE command, you only
>> do one scan rather than having a second scan per operation.  If that's
>> currently working, we probably don't want to make it stop working.
> 
> OK.
> 
> How about squash Jeevan's and my patch, so both
> check_default_allows_bound() and ValidatePartitionConstraints() know
> to scan default partition's children and there won't be any surprises
> in the regression test output as you found after applying just the
> Jeevan's patch.  Unfortunately, I'm not able to post such a patch
> right now.

I guess we don't need to squash, as they could be seen as implementing
different features. Reordering the patches helps though.  So, apply them
in this order:

1. My patch to teach ValidatePartitionConstraints() to skip scanning
   a partition's own partitions, which optimizes ATTACH PARTITION
   command's partition constraint validation scan (this also covers the
   case of scanning the default partition to validate its updated
   constraint when attaching a new partition)

2. Jeevan's patch to teach check_default_allows_bound() to skip scanning
   the default partition's own partitions, which covers the case of
   scanning the default partition to validate its updated constraint when
   adding a new partition using CREATE TABLE

Attached 0001 and 0002 are ordered that way.

In addition to implementing the features mentioned in 1 and 2 above, the
patches also modify the INFO message to mention "updated partition
constraint for default partition \"%s\"", instead of "partition constraint
for table \"%s\"", when the default partition is involved.  That's so that
it's consistent with the error message that would be emitted by either
check_default_allows_bound() or ATRewriteTable() when the scan finds a row
that violates updated default partition constraint, viz. the following:
"updated partition constraint for default partition \"%s\" would be
violated by some row"

Thoughts?

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Sep 26, 2017 at 4:27 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2017/09/16 1:57, Amit Langote wrote:
>> On Sat, Sep 16, 2017 at 12:59 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> I believe the intended advantage of the current system is that if you
>>> specify multiple operations in a single ALTER TABLE command, you only
>>> do one scan rather than having a second scan per operation.  If that's
>>> currently working, we probably don't want to make it stop working.
>>
>> OK.
>>
>> How about squash Jeevan's and my patch, so both
>> check_default_allows_bound() and ValidatePartitionConstraints() know
>> to scan default partition's children and there won't be any surprises
>> in the regression test output as you found after applying just the
>> Jeevan's patch.  Unfortunately, I'm not able to post such a patch
>> right now.
>
> I guess we don't need to squash, as they could be seen as implementing
> different features. Reordering the patches helps though.  So, apply them
> in this order:
>
> 1. My patch to teach ValidatePartitionConstraints() to skip scanning
>    a partition's own partitions, which optimizes ATTACH PARTITION
>    command's partition constraint validation scan (this also covers the
>    case of scanning the default partition to validate its updated
>    constraint when attaching a new partition)
>
> 2. Jeevan's patch to teach check_default_allows_bound() to skip scanning
>    the default partition's own partitions, which covers the case of
>    scanning the default partition to validate its updated constraint when
>    adding a new partition using CREATE TABLE
>
> Attached 0001 and 0002 are ordered that way.

OK, I pushed all of this, spread out over 3 commits.  I reworked the
test cases not to be entangled with the existing test cases, and also
to test both of these highly-related features together.  Hopefully you
like the result.

Thanks for the patches and the analysis.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/10/06 2:25, Robert Haas wrote:
> On Tue, Sep 26, 2017 at 4:27 AM, Amit Langote wrote:
>> I guess we don't need to squash, as they could be seen as implementing
>> different features. Reordering the patches helps though.  So, apply them
>> in this order:
>>
>> 1. My patch to teach ValidatePartitionConstraints() to skip scanning
>>    a partition's own partitions, which optimizes ATTACH PARTITION
>>    command's partition constraint validation scan (this also covers the
>>    case of scanning the default partition to validate its updated
>>    constraint when attaching a new partition)
>>
>> 2. Jeevan's patch to teach check_default_allows_bound() to skip scanning
>>    the default partition's own partitions, which covers the case of
>>    scanning the default partition to validate its updated constraint when
>>    adding a new partition using CREATE TABLE
>>
>> Attached 0001 and 0002 are ordered that way.
> 
> OK, I pushed all of this, spread out over 3 commits.  I reworked the
> test cases not to be entangled with the existing test cases, and also
> to test both of these highly-related features together.  Hopefully you
> like the result.

Thanks for committing.

I noticed that 6476b26115f3 doesn't use the same INFO message as
14f67a8ee28.  You can notice in the following example that the message
emitted (that the default partition scan is skipped) is different when a
new partition is added with CREATE TABLE and with ATTACH PARTITION,
respectively.

create table foo (a int, b char) partition by list (a);

-- default partition
create table food partition of foo default partition by list (b);

-- default partition's partition with the check constraint
create table fooda partition of food (check (not (a is not null and a = 1
and a = 2))) for values in ('a');

create table foo1 partition of foo for values in (1);
INFO:  partition constraint for table "fooda" is implied by existing
constraints
CREATE TABLE

create table foo2 (like foo);
alter table foo attach partition foo2 for values in (2);
INFO:  updated partition constraint for default partition "fooda" is
implied by existing constraints
ALTER TABLE

That's because it appears that you applied Jeevan's original patch.
Revised version of his patch that I had last posted contained the new
message. Actually the revised patch had not only addressed the case where
the default partition's child's scan is skipped, but also the case where
the scan of default partition itself is skipped.  As things stand now:

alter table food add constraint skip_check check (not (a is not null and
(a = any (array[1, 2]))));

create table foo1 partition of foo for values in (1);
INFO:  partition constraint for table "food" is implied by existing
constraints
CREATE TABLE

create table foo2 (like foo);
CREATE TABLE

alter table foo attach partition foo2 for values in (2);
INFO:  updated partition constraint for default partition "food" is
implied by existing constraints
ALTER TABLE


Attached a patch to modify the INFO messages in check_default_allows_bound.

Thanks,
Amit

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Oct 5, 2017 at 9:29 PM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> Attached a patch to modify the INFO messages in check_default_allows_bound.

Committed.  However, I didn't see a reason to adopt the comment change
you proposed, so I left that part out.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/10/13 4:18, Robert Haas wrote:
> On Thu, Oct 5, 2017 at 9:29 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Attached a patch to modify the INFO messages in check_default_allows_bound.
> 
> Committed.  However, I didn't see a reason to adopt the comment change
> you proposed, so I left that part out.

Okay, thanks.

Regards,
Amit



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers