BUG #19857: Stats restore rejects empty arrays, losing range and array stats

Поиск
Список
Период
Сортировка
Искать

BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
PG Bug reporting form <noreply@postgresql.org>
Дата:
The following bug has been logged on the website:

Bug reference:      19857
Logged by:          Atish Prakash
Email address:      125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system:   Linux x86_64
Description:        

If a range column has fewer than two non-empty values, ANALYZE stores an
empty
range length histogram. Restoring those statistics fails with a WARNING, so
range_empty_frac is lost too, and estimates change a lot.

    CREATE TABLE r(a int4range);
    INSERT INTO r SELECT 'empty' FROM generate_series(1, 10000);
    ANALYZE r;

Then dump with "pg_dump --statistics" and restore into an empty database:

    master:  WARNING:  "range_length_histogram" must be a one-dimensional
array
    18.6:    WARNING:  argument "range_length_histogram" must be specified
when
             argument "range_empty_frac" is specified (message shortened)

Estimates on master, with autovacuum disabled on the table so that a later
ANALYZE
does not replace the statistics:

    query                                   actual   before   after
    SELECT * FROM r WHERE a <@ '[1,2)'       10000    10000      89
    SELECT * FROM r WHERE a && '[1,2)'           0        1     178

A table with one non-empty value and 9,999 empty ones is affected the same
way
(estimate 9999 before, 87 after, for the first form of query, with actual
10000).
Multirange columns also produce the warning.

pg_upgrade loses them too: after an upgrade from 18.6 to 19rc1,
range_empty_frac for
table r went from 1 to NULL, and the first query above was estimated at 50
rows.

On 19rc1 and master the same check rejects most_common_elems when it is
empty, which
commit 261f89a976 makes common. An ordinary array column with all-distinct
elements
is enough, and tsvector columns are affected too:

    CREATE TABLE tags(id int, a int[]);
    INSERT INTO tags
        SELECT i, ARRAY[i, i + 1000000] FROM generate_series(1, 50000) i;
    ANALYZE tags;

    WARNING:  "most_common_elems" must be a one-dimensional array

    query                                   actual   before   after
    SELECT * FROM tags WHERE a @> '{5}'          1        3    ~270
    SELECT * FROM tags WHERE a && '{5,6}'        2        7    ~530

Extended statistics are hit harder on master. An empty array in one
expression's
statistics rejects the statistics for every expression in that object,
including
unrelated ones:

    WARNING:  could not parse "exprs": invalid element in expression -1

Expected: the statistics restore as they were, without a warning.

Cause: on master, pg_stats shows range_length_histogram as '{}' and pg_dump
emits
that. statatt_build_stavalues() requires a one-dimensional array, and an
empty
array has zero dimensions. On 18.6, pg_stats shows it as NULL, so pg_dump
omits it
while still emitting range_empty_frac, and the restore function rejects that
pair.

I have not written a patch.

Found with dump and restore differential tests. I reproduced and checked the
result myself.




Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:


On Tue, Nov 10, 2026 at 4:48 AM PG Bug reporting form <noreply@postgresql.org> wrote:
The following bug has been logged on the website:

Bug reference:      19857
Logged by:          Atish Prakash
Email address:      125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system:   Linux x86_64
Description:       

If a range column has fewer than two non-empty values, ANALYZE stores an
empty
range length histogram. Restoring those statistics fails with a WARNING, so
range_empty_frac is lost too, and estimates change a lot.

    CREATE TABLE r(a int4range);
    INSERT INTO r SELECT 'empty' FROM generate_series(1, 10000);
    ANALYZE r;

Then dump with "pg_dump --statistics" and restore into an empty database:

    master:  WARNING:  "range_length_histogram" must be a one-dimensional
array
    18.6:    WARNING:  argument "range_length_histogram" must be specified
when
             argument "range_empty_frac" is specified (message shortened)

Estimates on master, with autovacuum disabled on the table so that a later
ANALYZE
does not replace the statistics:

    query                                   actual   before   after
    SELECT * FROM r WHERE a <@ '[1,2)'       10000    10000      89
    SELECT * FROM r WHERE a && '[1,2)'           0        1     178

A table with one non-empty value and 9,999 empty ones is affected the same
way
(estimate 9999 before, 87 after, for the first form of query, with actual
10000).
Multirange columns also produce the warning.

pg_upgrade loses them too: after an upgrade from 18.6 to 19rc1,
range_empty_frac for
table r went from 1 to NULL, and the first query above was estimated at 50
rows.

On 19rc1 and master the same check rejects most_common_elems when it is
empty, which
commit 261f89a976 makes common. An ordinary array column with all-distinct
elements
is enough, and tsvector columns are affected too:

    CREATE TABLE tags(id int, a int[]);
    INSERT INTO tags
        SELECT i, ARRAY[i, i + 1000000] FROM generate_series(1, 50000) i;
    ANALYZE tags;

    WARNING:  "most_common_elems" must be a one-dimensional array

    query                                   actual   before   after
    SELECT * FROM tags WHERE a @> '{5}'          1        3    ~270
    SELECT * FROM tags WHERE a && '{5,6}'        2        7    ~530

Extended statistics are hit harder on master. An empty array in one
expression's
statistics rejects the statistics for every expression in that object,
including
unrelated ones:

    WARNING:  could not parse "exprs": invalid element in expression -1

Expected: the statistics restore as they were, without a warning.

Cause: on master, pg_stats shows range_length_histogram as '{}' and pg_dump
emits
that. statatt_build_stavalues() requires a one-dimensional array, and an
empty
array has zero dimensions. On 18.6, pg_stats shows it as NULL, so pg_dump
omits it
while still emitting range_empty_frac, and the restore function rejects that
pair.

I have not written a patch.

Found with dump and restore differential tests. I reproduced and checked the
result myself.


Looking into this, probably a duplicate of 19842
 

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:
Вложения:



Looking into this, probably a duplicate of 19842

It's definitely a duplicate of 19842, but because it would need to be backported to 18, we'd want to handle this one first, and build the 19842-stuff on top of that.

I have relaxed the test in stats_check_required_arg() to allow for empty arrays, which itself creates a burden of checking the number of dimensions of an array before asking for the number of vals in the 0th dimension, which needs to be done twice in attribute_statistics_update_internal() and nowhere else, mostly because there are only two arrays (most_common_values and most common_freqs) that need to have an equal number of values (MCELEM has additional values over its freqs, and range_length_hisotgram is paired with the scalar range_null_frac as a way to save slots, I presume.

Added additional test cases to show that the length agreement requirement still works on MCV/MCF when one is empty, and that both store empty arrays without complaint.

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:
Вложения:

Same patch, commit message updated to have both discussion threads.

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Michael Paquier <michael@paquier.xyz>
Дата:
Вложения:
On Tue, Nov 10, 2026 at 08:50:39PM -0500, Corey Huinker wrote:
> Same patch, commit message updated to have both discussion threads.

 	arr = DatumGetArrayTypeP(PG_GETARG_DATUM(argnum));
 
-	if (ARR_NDIM(arr) != 1)
+	if (ARR_NDIM(arr) > 1)
 	{
 		ereport(WARNING,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),

This change would trigger a CVE, I suspect..  Under valgrind, and for
example these sequences:
CREATE TABLE t (a int);
-- most_common_freqs
SELECT pg_restore_attribute_stats(
    'schemaname', 'public', 'relname', 't', 'attname', 'a',
    'inherited', false,
    'most_common_vals', '{}'::text,
    'most_common_freqs', '{}'::real[]);
EXPLAIN SELECT * FROM t WHERE a = 1;

CREATE TABLE t2 (b int[]);
-- most_common_elem_freqs
SELECT pg_restore_attribute_stats(
    'schemaname', 'public', 'relname', 't2', 'attname', 'b',
    'inherited', false,
    'most_common_elems', '{}'::text,
    'most_common_elem_freqs', '{}'::real[]);
EXPLAIN SELECT * FROM t2 WHERE b @> '{1}';
-- elem_count_histogram
SELECT pg_clear_attribute_stats('public', 't2', 'b', false);
SELECT pg_restore_attribute_stats(
    'schemaname', 'public', 'relname', 't2', 'attname', 'b',
    'inherited', false,
    'elem_count_histogram', '{}'::real[]);
EXPLAIN SELECT * FROM t2, unnest(b);

Then get_attstatsslot() does an out-of-bound read for
most_common_freqs, most_common_elem_freqs and elem_count_histogram
because an array with 0 dimension has no follow-up data as far as I
can see, giving this pattern:
==804366== Invalid read of size 4
==804366==    at 0x1A58924: get_attstatsslot (lsyscache.c:3636)
==804366==    by 0x1950B6C: var_eq_const (selfuncs.c:422)
==804366==    by 0x195063F: eqsel_internal (selfuncs.c:349)
==804366==    by 0x1950297: eqsel (selfuncs.c:302)
==804366==    by 0x1ACE474: FunctionCall4Coll (fmgr.c:1215)
==804366==    by 0x1AD0C37: OidFunctionCall4Coll (fmgr.c:1451)
==804366==    by 0x122CC7F: restriction_selectivity (plancat.c:2242)
==804366==    by 0x1063476: clause_selectivity_ext (clausesel.c:854)
==804366==    by 0x1060B84: clauselist_selectivity_ext (clausesel.c:136)
==804366==    by 0x1060AB1: clauselist_selectivity (clausesel.c:106)
==804366==    by 0x10823BD: set_baserel_size_estimates (costsize.c:5615)
==804366==    by 0x104B64A: set_plain_rel_size (allpaths.c:641)

Instead of allowing all these arrays to be empty, we should snipe at
range_length_histogram *and* most_common_elems, and only all these
two without generalizing the rule.  The other three are rejected
anyway at load, so blocking them at restore would be a consistency
move.

Perhaps we should update get_attstatsslot() as well to avoid the
over-read issue to begin with; that feels like a sloppy thing to do if
there is an empty array to process.

We'll likely need to do something similar for the extstats.  One step
at a time.  We could of course do a more relaxing approach like you
are proposing, but I cannot really get excited to let that go through
at restore time as the loads would fail even if get_attstatsslot() is
adjusted (actually this path should be adjusted anyway), and the
option-basis thing is not that invasive in the attached, at least it
seems so to me.

Attached is a cleaned-up patch.  So, thoughts or comments?
--
Michael

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:
On Wed, Nov 11, 2026 at 2:41 AM Michael Paquier <michael@paquier.xyz> wrote:
On Tue, Nov 10, 2026 at 08:50:39PM -0500, Corey Huinker wrote:
> Same patch, commit message updated to have both discussion threads.

        arr = DatumGetArrayTypeP(PG_GETARG_DATUM(argnum));

-       if (ARR_NDIM(arr) != 1)
+       if (ARR_NDIM(arr) > 1)
        {
                ereport(WARNING,
                                (errcode(ERRCODE_INVALID_PARAMETER_VALUE),

Nitpick: I think the error message itself needs to change based on allow_empty, as true allows 1-D or empty, and false allows only 1-D.
 
Instead of allowing all these arrays to be empty, we should snipe at
range_length_histogram *and* most_common_elems, and only all these
two without generalizing the rule.  The other three are rejected
anyway at load, so blocking them at restore would be a consistency
move.

I thought about that, and wanted to at least try this method first. I'm glad we could rule it out early.

Perhaps we should update get_attstatsslot() as well to avoid the
over-read issue to begin with; that feels like a sloppy thing to do if
there is an empty array to process.

We'll likely need to do something similar for the extstats.  One step
at a time.  We could of course do a more relaxing approach like you
are proposing, but I cannot really get excited to let that go through
at restore time as the loads would fail even if get_attstatsslot() is
adjusted (actually this path should be adjusted anyway), and the
option-basis thing is not that invasive in the attached, at least it
seems so to me.

The more relaxing approach just means a lot more NDIM-checks, which undoes the elegance of the initial fix.

This way seems not invasive at all, so I like it better.
 
Attached is a cleaned-up patch.  So, thoughts or comments?

I'll redo the ext stats patchset to this pattern. Do you intend to commit this soon? Asking so I know whether to rebase-away the 0001 patch.

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Michael Paquier <michael@paquier.xyz>
Дата:
Вложения:
On Wed, Nov 11, 2026 at 08:46:56AM -0500, Corey Huinker wrote:
> Nitpick: I think the error message itself needs to change based on
> allow_empty, as true allows 1-D or empty, and false allows only 1-D.

Yeah, point taken that this could be a little bit better, in a shape
similar to what you are suggesting.  I can see that we have similar
requirements in some of the array functions, though..  Will try to
shape that this way.

> This way seems not invasive at all, so I like it better.

Okay, noted.

> I'll redo the ext stats patchset to this pattern. Do you intend to commit
> this soon? Asking so I know whether to rebase-away the 0001 patch.

Yeah, I intend to wrap this one this morning.
--
Michael

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Michael Paquier <michael@paquier.xyz>
Дата:
Вложения:
On Thu, Nov 12, 2026 at 07:40:47AM +0900, Michael Paquier wrote:
> Yeah, I intend to wrap this one this morning.

Applied this down to v19, to match with the empty array cases that
have been introduced in 261f89a976b.

I think that this leaves an extra case, though...  In v17 and v18, it
is possible for range_empty_frac to be stored with
range_length_histogram as NULL.  On v19 and HEAD,
range_length_histogram is an empty array, so the two are always set.
The restore functions require the two together when upgrading to 18~,
meaning that some stats values could be lost when upgrading/dumping
the stats from a v17 or v18.

That's one pattern that Atish was mentioning originally in his report,
even if I got a bit lost in my mind due to the dump/upgrade mentions
with v19 and HEAD.

I cannot really get excited to allow this combination in the restore
functions, though, as it means being more lenient on
range_length_histogram forever, and the v19~ decision of using an
empty array is much better to me in the long term.  An extra argument
is that this case is super narrow anyway, unlikely to be hit in
practice?  As far as I can see, you need a [multi]range column with
less than *2* non-empty values..

I agree that we should move on with fixing the extstats restore
functions, of course, in terms of the handling of the empty array
values, to map now with what the attribute restores are able to do.
And fix this second issue down to v19.  Let's do that on the other
thread, we are done here for me.
--
Michael

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Michael Paquier <michael@paquier.xyz>
Дата:
Вложения:
On Thu, Nov 12, 2026 at 10:26:00AM +0900, Michael Paquier wrote:
> I cannot really get excited to allow this combination in the restore
> functions, though, as it means being more lenient on
> range_length_histogram forever, and the v19~ decision of using an
> empty array is much better to me in the long term.  An extra argument
> is that this case is super narrow anyway, unlikely to be hit in
> practice?  As far as I can see, you need a [multi]range column with
> less than *2* non-empty values..

Or we could do something on the dump side, forcing an empty array out
if seeing a NULL range_length_histogram..  Still not sure if that's
worth bothering, just saying.
--
Michael

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:
On Wed, Nov 11, 2026 at 8:32 PM Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Nov 12, 2026 at 10:26:00AM +0900, Michael Paquier wrote:
> I cannot really get excited to allow this combination in the restore
> functions, though, as it means being more lenient on
> range_length_histogram forever, and the v19~ decision of using an
> empty array is much better to me in the long term.  An extra argument
> is that this case is super narrow anyway, unlikely to be hit in
> practice?  As far as I can see, you need a [multi]range column with
> less than *2* non-empty values..

Or we could do something on the dump side, forcing an empty array out
if seeing a NULL range_length_histogram..  Still not sure if that's
worth bothering, just saying.
--
Michael

I had also considered fixing it dump-side as well, which seems cleaner but also means that the bug would persist in pre-existing dump files, though as you mention it may not be worth bothering because this is a corner case that only occurs if the histogram was less than 2 elements long, thus the query plan wouldn't be that far off. Also, such a pre-existing dump file would itself be restored to a cold-start database, and therefore unlikely to be getting flooded with queries before vacuumdb has had time to run in the way that a pg_upgrade-ed database would.

Commit looks good on master and REL_19_STABLE.

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:

Commit looks good on master and REL_19_STABLE.

But the problem still exists on 18, and that isn't a clean backport. Michael, do you want me to take that one?

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Michael Paquier <michael@paquier.xyz>
Дата:
Вложения:
On Wed, Nov 11, 2026 at 11:55:08PM -0500, Corey Huinker wrote:
> But the problem still exists on 18, and that isn't a clean backport.
> Michael, do you want me to take that one?

The only problem that can be reached is if we dump or upgrade from a
v17 or v18 instance with the pattern of a range_empty_frac set with
range_length_histogram as NULL, where we could perhaps switch
range_length_histogram to '{}' under-the-hood on the dump side if a
range_empty_frac exists.  I don't want the restore functions to live
eternally with a path authorizing for range_length_histogram=NULL if
there is a range_empty_frac, especially now that
range_length_histogram has to be an empty array since v19~.

And the end, I have no plans to bother based on the table definition
this requires.  I could still care with something on the dump side,
but I doubt that's worth the effort anyway.  What's been done on HEAD
and v19 is enough for these new empty-array cases.  :)
--
Michael

Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

От:
Corey Huinker <corey.huinker@gmail.com>
Дата:


On Thu, Nov 12, 2026 at 12:10 AM Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Nov 11, 2026 at 11:55:08PM -0500, Corey Huinker wrote:
> But the problem still exists on 18, and that isn't a clean backport.
> Michael, do you want me to take that one?

The only problem that can be reached is if we dump or upgrade from a
v17 or v18 instance with the pattern of a range_empty_frac set with
range_length_histogram as NULL, where we could perhaps switch
range_length_histogram to '{}' under-the-hood on the dump side if a
range_empty_frac exists.  I don't want the restore functions to live
eternally with a path authorizing for range_length_histogram=NULL if
there is a range_empty_frac, especially now that
range_length_histogram has to be an empty array since v19~.

And the end, I have no plans to bother based on the table definition
this requires.  I could still care with something on the dump side,
but I doubt that's worth the effort anyway.  What's been done on HEAD
and v19 is enough for these new empty-array cases.  :)
--
Michael

Sorry, I should have clarified - I was considering "18 can't import {}+empty_frac" a separate over the already-settled-not-gonna-do-it  "18 can't import NULL+empty_frac" issue, but your response above indicates even less appetite for allowing what would have to be synthetic stats just because that's the future valid value.

I'll make the v18-specific pg_dump fix and additional patch in the other thread.
FAQ