Обсуждение: Tracking notnull attributes inside Var

On Sun, May 15, 2022 at 8:41 AM Andy Fan <zhihui.fan1213@gmail.com> wrote:

>
> The var in RelOptInfo->reltarget should have nullable = 0 but the var in
> RelOptInfo->baserestrictinfo should have nullable = 1;  The beauty of this
> are: a). It can distinguish the two situations perfectly b). Whenever we want
> to know the nullable attribute of a Var for an expression, it is super easy to
> know. In summary, we need to maintain the nullable attribute at 2 different
> places. one is the before the filters are executed(baserestrictinfo, joininfo,
> ec_list at least).  one is after the filters are executed (RelOptInfo.reltarget
> only?)

Thanks for identifying this. What you have written makes sense and it
might open a few optimization opportunities. But let me put down some
other thoughts here. You might want to take those into consideration
when designing your solution.

Do we want to just track nullable and non-nullable. May be we want
expand this class to nullable (var may be null), non-nullable (Var is
definitely non-NULL), null (Var will be always NULL).

But the other way to look at this is along the lines of equivalence
classes. Equivalence classes record the expressions which are equal in
the final result of the query. The equivalence class members are not
equal at all the stages of query execution.  But because they are
equal in the final result, we can impose that restriction on the lower
levels as well. Can we think of nullable in that fashion? If a Var is
non-nullable in the final result, we can impose that restriction on
the intermediate stages since rows with NULL values for that Var will
be filtered out somewhere. Similarly we could argue for null Var. But
knowledge that a Var is nullable in the final result does not impose a
NULL, non-NULL restriction on the intermediate stages. If we follow
this thought process, we don't need to differentiate Var at different
stages in query.

>
> Come to JoinRel, we still need to maintain the 2 different cases as well.
>
> As for the joinrel.reltarget, currently it looks up the inputrel's reltarget to
> get the Var, so it is easy to inherit from Var->nullable from inputrel, but
> we need to consider the new  changes introduced by current join,
> Like new NOT nullable attributes because of join clauses OR new nullable
> attributes because of outer join.  Everything looks good for now.

Yes, if we want to maintain nullness at different stages in the query.

>
> The hard part is RelOptInfo.joininfo & root->eq_classes. All of them uses
> the shared RestrictInfo, and it is unclear which Var->nullable should be used in
> them. To not provide a wrong answer, I think we can assume nullable=-1 (unknown)
> and let the upper layer decides what to do (do we have known use cases to use
> the nullable attribute here?).

I think what applies to baserestrictinfo and reltarget also applies to
joininfo and join's reltarget. There will be three stages - join
clauses, join quals and reltarget.

In EQs the Vars in RestrictInfo will come from joininfo but EQ member
Vars will derive their nullable-ness from corresponding reltarget. I
can be wrong though.

>
> More considerations about this strategy:
> 1. We might use more memory for different var copies, the only known cases
>    RelOptInfo->reltarget for now.

When a Var is copied the whole expression tree needs to be copied.
That might be more memory than just copies of Var nodes.

> 2. _equalVar() has more complex semantics: shall we consider nulls or not.

This is interesting. It might have impact on set_plan_references and
planner's ability to search and match expressions.

But if we take the approach I have suggested earlier, this question
will not arise.

-- 
Best Wishes,
Ashutosh Bapat

Andy Fan <zhihui.fan1213@gmail.com> writes:
> notnulls discussion is forked from UniqueKey stuff, you can see the
> attachment
> for the UnqiueKey introduction. Tom raised his opinion to track the
> nullability
> inside Var[1][2][3], this thread would start from there based on my
> understanding.

I'm pretty certain that I never suggested this:

> struct Var
> {
> ...;
>       int nullable;  // -1 unknown,  0 - not nullable.  1 - nullable
> };

You're free to pursue it if you like, but I think it will be a dead end.
The fundamental problem as you note is that equalVar() cannot do anything
sane with a field defined that way.  Also, we'd have to generate Vars
initially with nullable = unknown (else, for example, ALTER SET/DROP NOT
NULL breaks stored views referring to the column).  It'd be on the planner
to run through the tree and replace that with "nullable" or "not
nullable".  It's hard to see how that's more advantageous than just
keeping the info in the associated RelOptInfo.

Also, I think you're confusing two related but distinct issues.  For
certain optimization issues, we'd like to keep track of whether a column
stored in a table is known NOT NULL.  However, that's not the same thing
as the question that I've muttered about, which is how to treat a Var
that's been possibly forced to null due to null-extension of an outer
join.  That is a different value from the Var as read from the table,
but we currently represent it the same within the planner, which causes
various sorts of undesirable complication.  We cannot fix that by setting
Var.nullable = true in above-the-join instances, because it might also
be true in below-the-join instances.  "Known not null in the table" is
not the inverse of "potentially nulled by an outer join".  Moreover, we
probably need to know *which* join is the one potentially nulling the Var,
so a bool is not likely enough anyway.

The schemes I've been toying with tend to look more like putting a
PlaceHolderVar-ish wrapper around the Var or expression that represents
the below-the-join value.  The wrapper node could carry any join ID
info that we find necessary.  The thing that I'm kind of stalled on is
how to define this struct so that it's not a big headache for join
strength reduction (which could remove the need for a wrapper altogether)
or outer-join reordering (which makes it a bit harder to define which
join we think is the one nulling the value).

            regards, tom lane