Re: ON EMPTY clause for aggregate and window functions

Поиск
Список
Период
Сортировка
Искать

Re: ON EMPTY clause for aggregate and window functions

От:
Tom Lane <tgl@sss.pgh.pa.us>
Дата:
Robert Haas  writes:
> This seems pretty useless -- the new syntax is more work for us to
> maintain, and doesn't really add any value over just using COALESCE.

It could add value, in scenarios where substitute-for-NULL doesn't
give quite the behavior you want.  But that's not so for the two
aggregates the spec has bothered to define this for, and I'm having
a hard time coming up with an aggregate for which it would be so.

If somebody actually did have an aggregate that could return null for
more than zero input rows, they could always do

    CASE WHEN count(x) > 0 THEN frobnitz(x) ELSE value_for_zero_rows END

which has the extra benefit that you can choose "count(x)" or
"count(*)" depending on your desires for what to do with null
inputs.

> To be clear, I expect to lose this argument on the grounds that
> apparently this syntax is in the spec and therefore we ought to
> support it. But I don't understand why the spec -- or we -- should
> spend time inventing new ways to spell existing behaviors. That just
> seems confusing (and this particular choice of syntax seems
> extra-confusing).

Yes, this choice of syntax sucks pretty badly.  In the COALESCE
spelling, it's crystal clear that the substitute value is not an
aggregate argument and so is evaluated at most once (per group);
there's no need for gamesmanship around restricting it to be a
constant.

I'd be totally fine with rejecting this as a frammish we do
not care to support.

			regards, tom lane


Re: ON EMPTY clause for aggregate and window functions

От:
Tom Lane <tgl@sss.pgh.pa.us>
Дата:
Vik Fearing  writes:
> On 17/09/2026 15:31, Tom Lane wrote:
>> I'd be totally fine with rejecting this as a frammish we do
>> not care to support.

> I am not the most unbiased person, but I would like us to implement it.
> If for nothing else, then for helping people convert from database 
> implementations that do have it.

It's not zero cost.  Robert already mentioned the
development/maintenance effort involved, and it also bloats the Bison
grammar rules, creating some incremental penalty on parsing speed.
Admittedly these costs aren't large, but neither is the benefit
of supporting it.

			regards, tom lane


Re: BUG #19638: Planner chooses an index-only scan for an index AM without amcanreturn, and execution fails

От:
Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Дата:
> Agreed, v3 fixed this.

Confirmed on 18.6. v3 applies with a 2-line offset and passes everything I
threw at the previous one:

    the #19638 reproducer                  Seq Scan (disabled) -> 3, correct
    count(*) over i_expr                       Index Only Scan (kept)
    count(*) over i_col                         Index Only Scan (unchanged)
    SELECT FROM t_nokey               Seq Scan (disabled), correct
    EXISTS (SELECT 1 FROM ...)     Seq Scan (disabled), correct
    make check                                   all 231 tests passed

Tracking any_canreturn inside the existing loop is nicer than the separate
pass I suggested -- one traversal, and it reads as part of building the
bitmap rather than as an afterthought.

Nothing further from me on this one.

El lun, 24 ago 2026 a las 15:34, Andrey Rachitskiy (<pl0h0yp1@gmail.com>) escribió:

пн, 24 авг. 2026 г. в 23:52, Manuel Reyes Bravo <manuelreyesbravo@gmail.com>:
    unpatched:  Aggregate -> Index Only Scan using i_expr on t_expr
    patched:    Aggregate -> Seq Scan on t_expr (disabled)

Agreed, v3 fixed this.

--
Regards,
Rachitskiy Andrey


--
Saludos cordiales,

Manuel Reyes

Re: BUG #19638: Planner chooses an index-only scan for an index AM without amcanreturn, and execution fails

От:
Manuel Reyes Bravo <manuelreyesbravo@gmail.com>
Дата:
Thanks for the patch. I tested it on REL_18_STABLE-equivalent sources (18.6,
built from source), since that is the branch this would need to be
back-patched to. Short version: it fixes the bug, it passes the full
regression suite, and I believe it also rejects a legitimate plan.

What works
----------

Applied cleanly (hunk offset 2 lines). With the test module:

    before:  Index Only Scan using i_nokey  -> ERROR: no data returned...
    after:   Seq Scan (disabled)            -> 3      correct

make check: all 231 tests passed.

What I think is a false positive
--------------------------------

The guard keys off bms_is_empty(index_canreturn_attrs), but that bitmapset
is
also empty for an index whose columns are all expressions, because the
loop
just above skips them:

    /*
     * For the moment, we just ignore index expressions.  It might be nice
     * to do something with them, later.
     */
    if (attno == 0)
        continue;

So "empty" does not mean "the AM can return nothing", it means "no plain
columns are returnable". A btree over an expression can feed an index-only
scan perfectly well. Measured on 18.6, with enable_seqscan off:

    CREATE TABLE t_expr (a int, b int);
    INSERT INTO t_expr SELECT g, g*2 FROM generate_series(1,50000) g;
    CREATE INDEX i_expr ON t_expr ((a + b));
    VACUUM ANALYZE t_expr;
    SELECT count(*) FROM t_expr;

    unpatched:  Aggregate -> Index Only Scan using i_expr on t_expr
    patched:    Aggregate -> Seq Scan on t_expr (disabled)

Both return 50000, so this is a plan regression rather than a correctness
one -- counting can no longer walk the smaller index. A control with an
ordinary column index (CREATE INDEX i_col ON t_col (a)) keeps its index-only
scan under the patch, so the effect is specific to expression-only indexes.

Worth noting: make check does not catch this. The suite passed 231/231 with
the patch applied, so this would go in unnoticed.

A variant that avoids it
------------------------

Attached as a patch this time, rather than an archive. It tests the AM's
capability directly instead of the bitmapset:

    if (result)
    {
        bool        any_canreturn = false;

        for (i = 0; i < index->ncolumns; i++)
        {
            if (index->canreturn[i])
            {
                any_canreturn = true;
                break;
            }
        }
        if (!any_canreturn)
            result = false;
    }

index->canreturn[] is filled per column from index_can_return() in
plancat.c,
including expression columns, so an expression btree has a true entry
while an
AM with amcanreturn == NULL has none.

Measured on 18.6 with that variant:

    the reproducer            -> 3, correct (bug fixed)
    count(*) over i_expr      -> Index Only Scan (no regression)
    count(*) over i_col       -> Index Only Scan (unchanged)
    make check                -> all 231 tests passed

The patch is against 18.6 sources, since that is what I tested on; it should
apply to master with an offset.

I have not tried to judge which shape you would prefer, and there may be a
reason to keep it keyed off the bitmapset that I am not seeing. I can rerun
any of this on 19beta2 as well if that is useful.

El lun, 24 ago 2026 a las 14:20, Andrey Rachitskiy (<pl0h0yp1@gmail.com>) escribió:


пн, 24 авг. 2026 г. в 21:51, Manuel Reyes Bravo <manuelreyesbravo@gmail.com>:

The attached tarball includes that script as alcance.sql.

In the future, it would be better to attach patches rather than archives.
I kept the fix minimal: one guard after bms_is_subset() in check_index_only(), rejecting the plan when no key column is returnable.

--
Regards,
Rachitskiy Andrey


--
Saludos cordiales,

Manuel Reyes

Re: ON EMPTY clause for aggregate and window functions

От:
Robert Haas <robertmhaas@gmail.com>
Дата:
On Sat, Sep 12, 2026 at 9:19 AM Jeevan Chalke
 wrote:
> ON EMPTY is now implemented as exactly:
>
>     agg(args, default ON EMPTY)  ==  COALESCE(agg(args), default)
>
> for both plain and window aggregates: compute the aggregate's ordinary
> result exactly as without ON EMPTY, and substitute the default only if that
> result is NULL. No new expression-evaluation step, no per-row bookkeeping,
> and no restriction on partial/parallel aggregation.

This seems pretty useless -- the new syntax is more work for us to
maintain, and doesn't really add any value over just using COALESCE.

To be clear, I expect to lose this argument on the grounds that
apparently this syntax is in the spec and therefore we ought to
support it. But I don't understand why the spec -- or we -- should
spend time inventing new ways to spell existing behaviors. That just
seems confusing (and this particular choice of syntax seems
extra-confusing).

-- 
Robert Haas


Re: BUG #19638: Planner chooses an index-only scan for an index AM without amcanreturn, and execution fails

От:
Andrey Rachitskiy <pl0h0yp1@gmail.com>
Дата:


пн, 24 авг. 2026 г. в 21:51, Manuel Reyes Bravo <manuelreyesbravo@gmail.com>:

The attached tarball includes that script as alcance.sql.

In the future, it would be better to attach patches rather than archives.
I kept the fix minimal: one guard after bms_is_subset() in check_index_only(), rejecting the plan when no key column is returnable.

--
Regards,
Rachitskiy Andrey

Re: BUG #19638: Planner chooses an index-only scan for an index AM without amcanreturn, and execution fails

От:
Andrey Rachitskiy <pl0h0yp1@gmail.com>
Дата:

пн, 24 авг. 2026 г. в 19:05, David Rowley <dgrowleyml@gmail.com>:
What do you mean by "required combination"?  We have plenty of
IndexAMs that don't implement amcanreturn, e.g. brin.c.

 
Dear David,

"No amcanreturn" alone is not enough.

The failure needs
amoptionalkey = true together with amgettuple != NULL and amcanreturn
NULL.  I could not find an in-core AM with that combination; diskann
has it, which is why the third-party extension showed the bug.

https://github.com/timescale/pgvectorscale/blob/main/pgvectorscale/src/access_method/mod.rs
```
amroutine.amoptionalkey = true;
...
amroutine.amgettuple = Some(scan::amgettuple);
amroutine.amgetbitmap = None;
```

I reproduced it on current master without pgvectorscale, using a tiny
module AM that only implements those flags and returns heap TIDs
without filling xs_hitup/xs_itup.  With enable_seqscan/bitmapscan off:
```
  Aggregate
    ->  Index Only Scan using idx_nopk on t_nopk

  ERROR:  no data returned for index-only scan
```
So this is not a mixed-binary problem with the extension.

The hole is in check_index_only(): bms_is_subset(attrs_used,
index_canreturn_attrs) is true when attrs_used is empty even if the
index cannot return any column.  That matches the empty-targetlist
count(*) case.  It is in the same family as the earlier
get_actual_variable_range() amcanreturn check (74197bdc842).
 

--
Regards,
Rachitskiy Andrey

Re: ON EMPTY clause for aggregate and window functions

От:
Jeevan Chalke <jeevan.chalke@enterprisedb.com>
Дата:
Hello Isaac,

Good points on both counts.

Regarding INITCOND, you are correct that it only sets the initial state.
I tried exploiting it in my very first attempt too.  However, if the
aggregate isn't invoked at all (zero input rows), INITCOND is never
used to determine the final result, which is why we need this mechanism
to return something at finalization when the input set was empty.
The majority of the code changes here are to determine whether we've
received any input or not.

I'm open to renaming 'default_value' if a better term comes up.
'Value_at_empty_input' is certainly more precise than 'default',
but it's long.  I'll keep an eye out for other suggestions; I'm not
attached to the current name and will be happy to rename it once we
settle on one.

Thanks for reviewing.

On Fri, Jun 26, 2026 at 6:51 PM Isaac Morland <isaac.morland@gmail.com> wrote:
On Fri, 26 Jun 2026 at 05:12, Jeevan Chalke <jeevan.chalke@enterprisedb.com> wrote:
Hello Hackers,

Here is a patch set adding an optional ON EMPTY clause to aggregate (and
aggregate-as-window-function) calls.  It supplies a value to return when the
aggregate processes no input rows at all:

    agg_function(args, default_value ON EMPTY)

For example:

    SELECT sum(i, -1 ON EMPTY)
    FROM generate_series(1,10) AS s(i) WHERE i > 100;
     sum
    -----
      -1
    (1 row)

ON EMPTY is triggered only by an empty input set, not by NULL inputs that are
ignored during aggregation.  A FILTER that removes all rows makes the input
set empty, so the default applies in that case too.  Because a grouped query
never produces empty groups, ON EMPTY takes effect for an ungrouped aggregate
over zero rows, or for a group whose rows are all removed by FILTER.  For an
aggregate used as a window function, the default is returned for any row whose
frame contains no rows.  It also works with an ordered-set aggregate, with the
default written before WITHIN GROUP:

Is there any chance of storing a default default_value with the aggregate? I ask because for most aggregate functions there is a specific value for each function which is almost always what will be wanted, e.g., 0 for sum, -Infinity for max, +Infinity for min, 1 for multiplication (if the other proposal for a multiplication aggregate is accepted), and so on, generally characterizable as the identity element for the function in question. Only in rare cases would one actually want to override the identity and use some other specified value. 

I was going to suggest there would need to be an additional clause for CREATE AGGREGATE, but I see there is already an INITCOND parameter which in principle should already be doing the job (except that as I understand it the aggregate isn't invoked at all for empty input?).

Also I'm not entirely happy with the name "default_value". It's not really a default, just the value of the aggregate at empty input. Unfortunately, I don't have a better suggestion.


--
Jeevan Chalke
Senior Principal Engineer, Engineering Manager
Product Development


enterprisedb.com

Re: ON EMPTY clause for aggregate and window functions

От:
Isaac Morland <isaac.morland@gmail.com>
Дата:
On Fri, 26 Jun 2026 at 05:12, Jeevan Chalke <jeevan.chalke@enterprisedb.com> wrote:
Hello Hackers,

Here is a patch set adding an optional ON EMPTY clause to aggregate (and
aggregate-as-window-function) calls.  It supplies a value to return when the
aggregate processes no input rows at all:

    agg_function(args, default_value ON EMPTY)

For example:

    SELECT sum(i, -1 ON EMPTY)
    FROM generate_series(1,10) AS s(i) WHERE i > 100;
     sum
    -----
      -1
    (1 row)

ON EMPTY is triggered only by an empty input set, not by NULL inputs that are
ignored during aggregation.  A FILTER that removes all rows makes the input
set empty, so the default applies in that case too.  Because a grouped query
never produces empty groups, ON EMPTY takes effect for an ungrouped aggregate
over zero rows, or for a group whose rows are all removed by FILTER.  For an
aggregate used as a window function, the default is returned for any row whose
frame contains no rows.  It also works with an ordered-set aggregate, with the
default written before WITHIN GROUP:

Is there any chance of storing a default default_value with the aggregate? I ask because for most aggregate functions there is a specific value for each function which is almost always what will be wanted, e.g., 0 for sum, -Infinity for max, +Infinity for min, 1 for multiplication (if the other proposal for a multiplication aggregate is accepted), and so on, generally characterizable as the identity element for the function in question. Only in rare cases would one actually want to override the identity and use some other specified value. 

I was going to suggest there would need to be an additional clause for CREATE AGGREGATE, but I see there is already an INITCOND parameter which in principle should already be doing the job (except that as I understand it the aggregate isn't invoked at all for empty input?).

Also I'm not entirely happy with the name "default_value". It's not really a default, just the value of the aggregate at empty input. Unfortunately, I don't have a better suggestion.

Re: ON EMPTY clause for aggregate and window functions

От:
Vik Fearing <vik@postgresfriends.org>
Дата:

On 17/09/2026 15:31, Tom Lane wrote:
> Yes, this choice of syntax sucks pretty badly.


I agree. The next edition isn't out yet, if we would like to suggest 
something better.


> I'd be totally fine with rejecting this as a frammish we do
> not care to support.


I am not the most unbiased person, but I would like us to implement it.  
If for nothing else, then for helping people convert from database 
implementations that do have it.

-- 

Vik Fearing



Re: ON EMPTY clause for aggregate and window functions

От:
Vik Fearing <vik@postgresfriends.org>
Дата:

On 12/09/2026 15:18, Jeevan Chalke wrote:
>
> ON EMPTY is now implemented as exactly:
>
> agg(args, default ON EMPTY)  ==  COALESCE(agg(args), default)


I've taken a quick look at this, and I found a few bugs.


1) The first one is that the constant requirement only looks for actual 
constants and not scoped constants.  For example:


CREATE TABLE cust (id INTEGER, name text, def_amount PRIMARY KEY (id));
CREATE TABLE ord (id INTEGER, custid INTEGER, amount INTEGER);
INSERT INTO cust SELECT g, 'c' || g, g FROM generate_series(1, 4) AS g (g);
INSERT INTO ord VALUES (1,1,100), (2,1,50), (3,3,7);

-- rejected: "ON EMPTY expression must be a constant value"
SELECT c.id,
        (SELECT SUM(o.amount, c.def_amount ON EMPTY)
         FROM ord AS o
         WHERE o.custid = c.id)
FROM cust AS c;


Here, the c.def_amount is constant for the subquery and should be 
accepted.  The example is perhaps a bit contrived, but the logic is sound.


2) Another bug I found is this:


CREATE TABLE mm (a INTEGER);
INSERT INTO mm SELECT g FROM generate_series(1, 10_000) AS g (g);
ANALYZE mm;

SELECT COALESCE(MAX(a), -1) FROM mm WHERE a > 100_000;  --  -1
SELECT MAX(a, -1 ON EMPTY)  FROM mm WHERE a > 100_000;  --  -1

CREATE INDEX ON mm (a);

SELECT COALESCE(MAX(a), -1) FROM mm WHERE a > 100_000;  -- -1
SELECT MAX(a, -1 ON EMPTY)  FROM mm WHERE a > 100_000;  --  NULL

When MAX and MIN get optimized with an index, the ON EMPTY seems to be 
dropped.


3) The set quantifier is not recognized.


SELECT SUM(ALL      a, 0 ON EMPTY) FROM t;
SELECT SUM(DISTINCT a, 0 ON EMPTY) FROM t;


Neither of those parse.


I will keep reviewing this feature.

-- 

Vik Fearing



FAQ