Обсуждение: [PATCH] fix DROP OPERATOR to reset links to itself on commutator and negator

Hello, hackers!

As was mentioned in that message [1], and earlier in [2], DROPping OPERATOR with
negator or commutator doesn't update respective fields (oprnegate and oprcom) on
them. While this issue is probably not very frequent, this behaviour leaves
database in inconsistent state and prevents from using former negator or
commutator.

Proposed patch fixes unwanted behaviour by reseting oprnegate and oprcom fields
to InvalidOid when deleting operator. Code updates tries to update other
operators only if necessary, and only if negator or commutator have target oid
in their respective fields.

To implement desired behaviour with as little code as possible, OperatorUpd
function is extended to accept additional parameter and made externally visible.
It's used from RemoveOperatorById, after checking that update is indeed needed.

Regression tests are added to check DROP OPERATOR behaves as intended (including
case with self-commutator and unlikely case with operator being both negator and
commutator).

Should this patch be added to CommitFest?

[1] — http://www.postgresql.org/message-id/CA+TgmoYThtZZf6yhnq22SZPw7OTiT68iu9wvvidb2Jz50KdAnQ@mail.gmail.com
[2] — http://www.postgresql.org/message-id/15208.1285772173@sss.pgh.pa.us

Cheers,
Roma

On 26-02-2016 12:46, Roma Sokolov wrote:
> Regression tests are added to check DROP OPERATOR behaves as intended (including
> case with self-commutator and unlikely case with operator being both negator and
> commutator).
> 
I don't think those are mandatory.

> Should this patch be added to CommitFest?
> 
Why not?

I didn't test your patch but

+  if (isDelete ? (t->oprcom == baseId || t->oprnegate == baseId)
+          : !OidIsValid(t->oprcom) || !OidIsValid(t->oprnegate))

... is hard to understand. Instead, you could separate the conditional
expression into a variable.

+ if (isDelete ? t->oprnegate == baseId : !OidIsValid(t->oprnegate))

It could be separate into a variable to be readable (or at least deserve
a comment).

(isDelete ? InvalidOid : ObjectIdGetDatum(baseId))

... and this one too. It is used in 4 places in that function.


--   Euler Taveira                   Timbira - http://www.timbira.com.br/  PostgreSQL: Consultoria, Desenvolvimento,
Suporte24x7 e Treinamento
 

Thanks for comments, Euler!

> ... is hard to understand. Instead, you could separate the conditional
> expression into a variable.

Fixed the patch to be more descriptive and to avoid repeating same
computation over and over again. See v2 of the patch attached.

> I don't think those are mandatory.


Why do you think that? Should I remove them or maybe send as separate
patch?

Cheers,
Roma

On 26-02-2016 17:51, Roma Sokolov wrote:
> Fixed the patch to be more descriptive and to avoid repeating same 
> computation over and over again. See v2 of the patch attached.
> 
Oh, much better.

> Why do you think that? Should I remove them or maybe send as separate
> patch?
> 
Because it is not a common practice to test catalog dependency on
separate tests (AFAICS initial catalogs are tested with oidjoins.sql).
Also, your test case is too huge for such a small use case. If you can
reduce it to a small set of commands using some of the oidjoins.sql
queries, I think it could be sufficient.


--   Euler Taveira                   Timbira - http://www.timbira.com.br/  PostgreSQL: Consultoria, Desenvolvimento,
Suporte24x7 e Treinamento
 

On Sat, Feb 27, 2016 at 12:46 AM, Roma Sokolov <sokolov.r.v@gmail.com> wrote:
> Should this patch be added to CommitFest?

Yes please.
-- 
Michael

Roma Sokolov wrote:
> See v2 of the patch attached.

Hello.
I have a stylistic comments. Sometimes you forget a space:
+  replaces[Anum_pg_operator_oprcom - 1] =true;

or use tab insted space:
+    if (OidIsValid(op->oprnegate) ||
+        (OidIsValid(op->oprcom)    && operOid != op->oprcom))
+        OperatorUpd(operOid,
+                    operOid    == op->oprcom ?    InvalidOid : op->oprcom,
+                    op->oprnegate,
+                    true);

And I think if you make this logic into a separate function,
it is possible to simplify the code. OperatorUpd function is too complex.

Also better to add comments to the tests.
The rest seems good.

PS I here thought it would be possible to print operators that have been
changed?

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

> On 27 Feb 2016, at 03:46, Euler Taveira <euler@timbira.com.br> wrote:
> Because it is not a common practice to test catalog dependency on
> separate tests (AFAICS initial catalogs are tested with oidjoins.sql).
> Also, your test case is too huge for such a small use case.

It seems oidjoins.sql is automatically generated and contains tests only for
initial database state. On the other hand, there are tests for CREATE OPERATOR
and ALTER OPERATOR, so it seems reasonable to me to have separate DROP OPERATOR
test, or to move all operator related testing to one file. This is however
clearly outside of the scope of this patch, so in v3 I've simplified tests using
queries from oidjoins.sql.

Cheers,
Roma

Hi Kevin,

On 3/1/16 11:08 AM, Roma Sokolov wrote:
>> On 27 Feb 2016, at 03:46, Euler Taveira <euler@timbira.com.br> wrote:
>> Because it is not a common practice to test catalog dependency on
>> separate tests (AFAICS initial catalogs are tested with oidjoins.sql).
>> Also, your test case is too huge for such a small use case. 
> 
> It seems oidjoins.sql is automatically generated and contains tests only for
> initial database state. On the other hand, there are tests for CREATE OPERATOR
> and ALTER OPERATOR, so it seems reasonable to me to have separate DROP OPERATOR
> test, or to move all operator related testing to one file. This is however
> clearly outside of the scope of this patch, so in v3 I've simplified tests using
> queries from oidjoins.sql.

You've signed up to review this patch, do you have an idea of when you
might be able to do the review?

Thanks,
-- 
-David
david@pgmasters.net

Hi,

On 03/01/2016 05:08 PM, Roma Sokolov wrote:
>> On 27 Feb 2016, at 03:46, Euler Taveira <euler@timbira.com.br> wrote:
>> Because it is not a common practice to test catalog dependency on
>> separate tests (AFAICS initial catalogs are tested with oidjoins.sql).
>> Also, your test case is too huge for such a small use case.
>
> It seems oidjoins.sql is automatically generated and contains tests only for
> initial database state. On the other hand, there are tests for CREATE OPERATOR
> and ALTER OPERATOR, so it seems reasonable to me to have separate DROP OPERATOR
> test, or to move all operator related testing to one file. This is however
> clearly outside of the scope of this patch, so in v3 I've simplified tests using
> queries from oidjoins.sql.

I think it's OK to have a separate regression test for this. Clearly 
oidjoins is not a good place for such tests, and as you point out there 
are already tests for CREATE/ALTER OPERATOR, so it's perfectly natural 
to add a new file for DROP OPERATOR. I don't think it contradicts any 
common practice, as the existing CREATE/ALTER OPERATOR tests do pretty 
much the same thing.

One comment for the tests, though - you're using a mix of tabs and 
spaces for indentation, which breaks psql unpredictably (when debugging 
and pasting the commands to psql). Which is a bit annoying.

A few more comments:

1) OperatorUpd (pg_operator.c)
------------------------------
  /*   * check and update the commutator & negator, if necessary   *   * We need a CommandCounterIncrement here in case
ofa self-commutator   * operator: we'll need to update the tuple that we just inserted.   */  if (!isDelete)
CommandCounterIncrement();

This would really deserve an explanation of why we don't need to 
increment the command counter for a delete.
  /* When deleting, reset other operator field to InvalidOid, otherwise,   * set it to point to operator being updated
*/
 

This comment is a bit broken - the first line should be just '/*' and 
the second line uses spaces instead of a tabulator.

I have to admit I find the existing code a bit convoluted, particularly 
the part that deals with the (commId == negId) case. And the patch does 
not really improve the situation, quite the contrary.

Perhaps it's time to get rid of this optimization? I mean, how likely it 
is to have an operator with the same negator and commutator? And how 
often we do DROP OPERATOR? Apparently even the original author doubted 
this, according to the comment right in front of the block.

2) operatorcmds.c
------------------

/* * Reset links on commutator and negator. Only do that if either * oprcom or oprnegate is set and given operator is
notself-commutator. * For self-commutator with negator prevent meaningful updates of the * same tuple by sending
InvalidOid.*/
 
if (OidIsValid(op->oprnegate) ||    (OidIsValid(op->oprcom) && operOid != op->oprcom))    OperatorUpd(operOid,
 operOid == op->oprcom ? InvalidOid : op->oprcom,            op->oprnegate,            true);
 

Firstly, this block contains tabulators within both the comment and the 
code (e.g. "is not" or in front of the "&&" operator. That seems a bit 
broken, I guess.

Also, maybe I'm missing something obvious, but it's not immediately 
obvious to me why we're only checking oprcom and not oprnegate? I.e. why 
shouldn't the code be

if (OidIsValid(op->oprnegate) ||    (OidIsValid(op->oprcom) && operOid != op->oprcom) ||    (OidIsValid(op->oprnegate)
&&operOid != op->oprnegate))    OperatorUpd(operOid,            operOid == op->oprcom ? InvalidOid : op->oprcom,
   operOid == op->oprnegate ? InvalidOid : op->oprnegate,            true);
 


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

Tomas, thanks for review and comments!

> I have to admit I find the existing code a bit convoluted, particularly the part that deals with the (commId ==
negId)case. And the patch does not really improve the situation, quite the contrary. 
>
> Perhaps it’s time to get rid of this optimization?

Indeed, code in OperatorUpd is not very easy to read, due to handling this case.
However we can achieve the same results without too much duplication. I have
changed OperatorUpd to perform tuple modification in "lazy" way. Please, check
it out in v4.patch (attached).

> Also, maybe I'm missing something obvious, but it's not immediately obvious to me why we're only checking oprcom and
notoprnegate? I.e. why shouldn’t the code be 

We do not need to check for operOid != op->oprnegate, since we can't create
operator that is negator to itself. Thus, opnergate either present and differs
from operator being deleted, or is InvalidOid. I have added some clarification
in the comment for future readers.


Fixed style issues as well.

Cheers,
Roma

Hi,

On 03/23/2016 12:50 AM, Roma Sokolov wrote:
> Hi,
>
> Tomas, thanks for review and comments!
>
>> I have to admit I find the existing code a bit convoluted,
>> particularly the part that deals with the (commId == negId) case.
>> And the patch does not really improve the situation, quite the
>> contrary. Perhaps it’s time to get rid of this optimization?
>
> Indeed, code in OperatorUpd is not very easy to read, due to
> handling this case. However we can achieve the same results without
> too much duplication. I have changed OperatorUpd to perform tuple
> modification in "lazy" way. Please, check it out in v4.patch
> (attached).

OK, the new code seems more comprehensible to me.

>> Also, maybe I'm missing something obvious, but it's not
>> immediately obvious to me why we're only checking oprcom and not
>> oprnegate? I.e. why shouldn’t the code be
>
> We do not need to check for operOid != op->oprnegate, since we can't
> create operator that is negator to itself. Thus, opnergate either
> present and differs from operator being deleted, or is InvalidOid. I
> have added some clarification in the comment for future readers.

Ah, I see. Thanks for the clarification.

> Fixed style issues as well.

I've noticed some whitespace issues in the OperatorUpd function - there 
are two or three lines with just a tabulator at the beginning, and one 
comment mixes indentation by tabs with spaces.

Also, it's generally recommended no to tweak formatting when not 
necessary, so perhaps don't remove the empty line at the end of the 
function (before simple_heap_update).

I think the comments will need rewording, but I'll leave that to a 
native speaker.

regards
Tomas

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Mar 22, 2016 at 9:25 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> OK, the new code seems more comprehensible to me.

I did not find this version very clear.  It wasn't consistent about
using ObjectIdGetDatum() where needed, but the bigger problem was that
I found the logic unnecessarily convoluted.  I rewrote it - I believe
more straightforwardly - as attached.  How does this look?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:
> I did not find this version very clear.  It wasn't consistent about
> using ObjectIdGetDatum() where needed, but the bigger problem was that
> I found the logic unnecessarily convoluted.  I rewrote it - I believe
> more straightforwardly - as attached.  How does this look?

I'd suggest that we save some code by always doing separate updates for
the commutator and negator entries.  We can handle the corner case where
they're the same by doing a CommandCounterIncrement between the updates,
instead of having convoluted and probably-never-yet-tested logic.

I'm also a bit dubious of the assumption in RemoveOperatorById that an
operator can't be its own negator.  Yeah, that should not be the case,
but if it is the case the deletion will fail outright.

We could resolve both of these issues by changing the semantics of
OprUpdate so that it unconditionally does a CommandCounterIncrement
after each update that it performs.  IMO that would be a lot simpler
and more bulletproof; it'd allow removal of a lot of these
overly-tightly-reasoned cases.
        regards, tom lane

On Wed, Mar 23, 2016 at 12:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> I did not find this version very clear.  It wasn't consistent about
>> using ObjectIdGetDatum() where needed, but the bigger problem was that
>> I found the logic unnecessarily convoluted.  I rewrote it - I believe
>> more straightforwardly - as attached.  How does this look?
>
> I'd suggest that we save some code by always doing separate updates for
> the commutator and negator entries.  We can handle the corner case where
> they're the same by doing a CommandCounterIncrement between the updates,
> instead of having convoluted and probably-never-yet-tested logic.

Sure, we could do that, but it isn't necessary.  If the logic never
gets hit, the question of whether it has bugs isn't that important.
And I'd rather not rejigger things more than necessary in something
that's going to be back-patched.

> I'm also a bit dubious of the assumption in RemoveOperatorById that an
> operator can't be its own negator.  Yeah, that should not be the case,
> but if it is the case the deletion will fail outright.

So what?  We've never guaranteed that things are going to work if you
start by corrupting the catalogs, and I wouldn't pick this as a place
to start.

> We could resolve both of these issues by changing the semantics of
> OprUpdate so that it unconditionally does a CommandCounterIncrement
> after each update that it performs.  IMO that would be a lot simpler
> and more bulletproof; it'd allow removal of a lot of these
> overly-tightly-reasoned cases.

I tried this, but it did not seem to work.  With the command counter
increments added and the conditional logic removed, I got:

rhaas=# CREATE OPERATOR === (PROCEDURE = int8eq, LEFTARG = bigint,
RIGHTARG = bigint);
CREATE OPERATOR
rhaas=# update pg_operator set oprnegate = oid where oprname = '===';
UPDATE 1
rhaas=# drop operator === (bigint, bigint);
ERROR:  attempted to delete invisible tuple

The same test case without those changes fails with:

ERROR:  tuple already updated by self

Interestingly, that test case passes on unpatched master.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Mar 23, 2016 at 12:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm also a bit dubious of the assumption in RemoveOperatorById that an
>> operator can't be its own negator.  Yeah, that should not be the case,
>> but if it is the case the deletion will fail outright.

> So what?  We've never guaranteed that things are going to work if you
> start by corrupting the catalogs, and I wouldn't pick this as a place
> to start.

I would not be worried except that it breaks a case that used to work,
as your test below demonstrates.

>> We could resolve both of these issues by changing the semantics of
>> OprUpdate so that it unconditionally does a CommandCounterIncrement
>> after each update that it performs.  IMO that would be a lot simpler
>> and more bulletproof; it'd allow removal of a lot of these
>> overly-tightly-reasoned cases.

> I tried this, but it did not seem to work.

Odd.  If you post the revised patch, I'll try to chase down what's wrong.
        regards, tom lane

I wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Wed, Mar 23, 2016 at 12:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> We could resolve both of these issues by changing the semantics of
>>> OprUpdate so that it unconditionally does a CommandCounterIncrement
>>> after each update that it performs.  IMO that would be a lot simpler
>>> and more bulletproof; it'd allow removal of a lot of these
>>> overly-tightly-reasoned cases.

>> I tried this, but it did not seem to work.

> Odd.  If you post the revised patch, I'll try to chase down what's wrong.

After playing with this, I'll bet you forgot that RemoveOperatorById would
need to re-fetch the target tuple if it got updated.  We could
alternatively fix that by skipping updates on the tuple due to be deleted,
but that would convolute the logic in OperatorUpd, which didn't seem
worthwhile to me.

I found some other stuff needing fixing (mostly typos in comments) and
also realized that we don't really need to bother with heap_modify_tuple
at all.  I pushed it with those fixes.
        regards, tom lane

On Fri, Mar 25, 2016 at 12:39 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Wed, Mar 23, 2016 at 12:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> We could resolve both of these issues by changing the semantics of
>>>> OprUpdate so that it unconditionally does a CommandCounterIncrement
>>>> after each update that it performs.  IMO that would be a lot simpler
>>>> and more bulletproof; it'd allow removal of a lot of these
>>>> overly-tightly-reasoned cases.
>
>>> I tried this, but it did not seem to work.
>
>> Odd.  If you post the revised patch, I'll try to chase down what's wrong.
>
> After playing with this, I'll bet you forgot that RemoveOperatorById would
> need to re-fetch the target tuple if it got updated.  We could
> alternatively fix that by skipping updates on the tuple due to be deleted,
> but that would convolute the logic in OperatorUpd, which didn't seem
> worthwhile to me.
>
> I found some other stuff needing fixing (mostly typos in comments) and
> also realized that we don't really need to bother with heap_modify_tuple
> at all.  I pushed it with those fixes.

Cool.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company