Обсуждение: BUG #15160: planner overestimates number of rows in join when thereare more than 200 rows coming from CTE
BUG #15160: planner overestimates number of rows in join when thereare more than 200 rows coming from CTE
От
PG Bug reporting form
Дата:
The following bug has been logged on the website:
Bug reference: 15160
Logged by: Alexey Ermakov
Email address: alexey.ermakov@dataegret.com
PostgreSQL version: 10.3
Operating system: Linux
Description:
Hello,
I'm wondering how planner estimates number of rows in that case:
create table test_in (id int primary key);
insert into test_in select id from generate_series(1,1000000) gs(id);
analyze test_in;
explain analyze with ids as (select id from generate_series(1,1000) gs(id)
limit 200)
select * from test_in where id in (select id from ids);
-------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=6.93..139.79 rows=200 width=4) (actual time=0.129..0.400
rows=200 loops=1)
CTE ids
-> Limit (cost=0.00..2.00 rows=200 width=4) (actual time=0.050..0.066
rows=200 loops=1)
-> Function Scan on generate_series gs (cost=0.00..10.00
rows=1000 width=4) (actual time=0.050..0.057 rows=200 loops=1)
-> HashAggregate (cost=4.50..6.50 rows=200 width=4) (actual
time=0.117..0.133 rows=200 loops=1)
Group Key: ids.id
-> CTE Scan on ids (cost=0.00..4.00 rows=200 width=4) (actual
time=0.051..0.086 rows=200 loops=1)
-> Index Only Scan using test_in_pkey on test_in (cost=0.42..0.66
rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=200)
Index Cond: (id = ids.id)
Heap Fetches: 200
Planning time: 0.128 ms
Execution time: 0.434 ms
explain analyze with ids as (select id from generate_series(1,1000) gs(id)
limit 201)
select * from test_in where id in (select id from ids);
-------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=6.96..132.78 rows=500000 width=4) (actual
time=0.119..0.389 rows=201 loops=1)
CTE ids
-> Limit (cost=0.00..2.01 rows=201 width=4) (actual time=0.048..0.064
rows=201 loops=1)
-> Function Scan on generate_series gs (cost=0.00..10.00
rows=1000 width=4) (actual time=0.048..0.056 rows=201 loops=1)
-> HashAggregate (cost=4.52..6.52 rows=200 width=4) (actual
time=0.113..0.130 rows=201 loops=1)
Group Key: ids.id
-> CTE Scan on ids (cost=0.00..4.02 rows=201 width=4) (actual
time=0.049..0.083 rows=201 loops=1)
-> Index Only Scan using test_in_pkey on test_in (cost=0.42..0.66
rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=201)
Index Cond: (id = ids.id)
Heap Fetches: 201
Planning time: 0.068 ms
Execution time: 0.417 ms
please note that it first example we got correct estimate of total number of
rows - 200, but in last one where CTE returned 201 rows (instead of 200) we
estimate total number of rows as 500000 (half of the table test_in).
which is way off and could lead to non optimal plan and poor performance.
I have same estimate if I replace IN clause with equivalent EXISTS subquery;
normal join estimates number of rows fine (but it's not equivalent in
general case when table_in.id is not unique).
reproduced in 9.5, 9.6 and 10. interesting thing that in postgresql 10
threshold is 200 rows but in previous version it's 199.
I suspect selectivity 0.5 we somehow get inside
compute_semi_anti_join_factors function in costsize.c but I'm not sure.
Thanks,
Alexey Ermakov
=?utf-8?q?PG_Bug_reporting_form?= <noreply@postgresql.org> writes:
> I'm wondering how planner estimates number of rows in that case:
See eqjoinsel_semi, particularly the change in behavior when it thinks
nd2 is or is not a default estimate.
Given the lack of statistics about the output of the WITH clause,
it's hard to see how we'd ever get trustworthy estimates here.
I think the fact that your first example yields an accurate
estimate is mostly luck.
regards, tom lane
On Tue, Apr 17, 2018 at 5:15 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
PG Bug reporting form <noreply@postgresql.org> writes:
> I'm wondering how planner estimates number of rows in that case:
See eqjoinsel_semi, particularly the change in behavior when it thinks
nd2 is or is not a default estimate.
Given the lack of statistics about the output of the WITH clause,
it's hard to see how we'd ever get trustworthy estimates here.
I think the fact that your first example yields an accurate
estimate is mostly luck.
regards, tom lane
There are similar issue without CTE which look pretty weird:
Good case with LIMIT 199 and adequate estimation:
hh=# explain SELECT * FROM resume WHERE resume_id IN (select id from generate_series(1, 1000) gs(id) LIMIT 199);
QUERY PLAN
---------------------------------------------------------------------------------------------------
Nested Loop (cost=21.53..108.98 rows=199 width=519)
-> Unique (cost=21.42..21.62 rows=199 width=4)
-> Sort (cost=21.42..21.52 rows=199 width=4)
Sort Key: gs.id
-> Limit (cost=0.00..9.95 rows=199 width=4)
-> Function Scan on generate_series gs (cost=0.00..50.00 rows=1000 width=4)
-> Index Scan using resume_pk on resume (cost=0.11..0.39 rows=1 width=519)
Index Cond: (resume_id = gs.id)
QUERY PLAN
---------------------------------------------------------------------------------------------------
Nested Loop (cost=21.53..108.98 rows=199 width=519)
-> Unique (cost=21.42..21.62 rows=199 width=4)
-> Sort (cost=21.42..21.52 rows=199 width=4)
Sort Key: gs.id
-> Limit (cost=0.00..9.95 rows=199 width=4)
-> Function Scan on generate_series gs (cost=0.00..50.00 rows=1000 width=4)
-> Index Scan using resume_pk on resume (cost=0.11..0.39 rows=1 width=519)
Index Cond: (resume_id = gs.id)
Very bad case with awful estimation (only difference LIMIT 200 vs LIMIT 199):
explain SELECT * FROM resume WHERE resume_id IN (select id from generate_series(1, 1000) gs(id) LIMIT 200);
QUERY PLAN
---------------------------------------------------------------------------------------------------
Nested Loop (cost=21.64..109.53 rows=45860504 width=519)
-> Unique (cost=21.53..21.73 rows=200 width=4)
-> Sort (cost=21.53..21.63 rows=200 width=4)
Sort Key: gs.id
-> Limit (cost=0.00..10.00 rows=200 width=4)
-> Function Scan on generate_series gs (cost=0.00..50.00 rows=1000 width=4)
-> Index Scan using resume_pk on resume (cost=0.11..0.39 rows=1 width=519)
Index Cond: (resume_id = gs.id)
QUERY PLAN
---------------------------------------------------------------------------------------------------
Nested Loop (cost=21.64..109.53 rows=45860504 width=519)
-> Unique (cost=21.53..21.73 rows=200 width=4)
-> Sort (cost=21.53..21.63 rows=200 width=4)
Sort Key: gs.id
-> Limit (cost=0.00..10.00 rows=200 width=4)
-> Function Scan on generate_series gs (cost=0.00..50.00 rows=1000 width=4)
-> Index Scan using resume_pk on resume (cost=0.11..0.39 rows=1 width=519)
Index Cond: (resume_id = gs.id)
It's not a problem by itself but once you start using this query with more joined tables - a lot bad things happens because 5 orders of magnitude error in selectivity estimation.
PS: in reality it forces us to use not more than 199 LIMIT in complex joins for batch operations or the database start generate funny plans.
Regards,
Maxim
--
Maxim Boguk
Senior Postgresql DBA
https://dataegret.com/
Phone RU: +7 985 433 0000
Phone UA: +380 99 143 0000
Phone AU: +61 45 218 5678
LinkedIn: http://www.linkedin.com/pub/maksym-boguk/80/b99/b1b
Skype: maxim.boguk
"Доктор, вы мне советовали так не делать, но почему мне по-прежнему больно когда я так делаю ещё раз?"
Senior Postgresql DBA
https://dataegret.com/
Phone RU: +7 985 433 0000
Phone UA: +380 99 143 0000
Phone AU: +61 45 218 5678
LinkedIn: http://www.linkedin.com/pub/maksym-boguk/80/b99/b1b
Skype: maxim.boguk
"Доктор, вы мне советовали так не делать, но почему мне по-прежнему больно когда я так делаю ещё раз?"
Maxim Boguk <maxim.boguk@gmail.com> writes: > On Tue, Apr 17, 2018 at 5:15 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> =?utf-8?q?PG_Bug_reporting_form?= <noreply@postgresql.org> writes: >>> I'm wondering how planner estimates number of rows in that case: >> See eqjoinsel_semi, particularly the change in behavior when it thinks >> nd2 is or is not a default estimate. > There are similar issue without CTE which look pretty weird: Yeah, this is exactly the same case as Alexey's: as soon as eqjoinsel_semi decides it's dealing with a default ndistinct estimate, it chickens out and delivers a very middle-of-the-road selectivity (0.5, it looks like). It's somewhat luck that the non-default path is giving you an accurate estimate, but certainly there's no surprise in the default case being way off. I don't particularly want to make that logic more aggressive about assuming it's calculating something real. The existing behavior was put in to fix a clear bug in the other direction, see https://www.postgresql.org/message-id/flat/201104112029.14738.uwe%40oss4u.com However, while looking at this I had a bit of an epiphany. The inner-join selectivity in the same cases is pretty much on-target, so is there any way we could factor that in? Yes, there is: the size of the semijoin output could not be more than the output of a plain inner join of the same two relations. So it'd be legitimate to clamp our selectivity estimate for the semijoin case to make it not more than the inner-join estimate. A little bit of hacking later, I have the attached patch. The bulk of the patch is just refactoring to avoid repetitive information lookup when we call both eqjoinsel_semi and eqjoinsel_inner. The actual change is just to clamp eqjoinsel_semi's result, like this: /* * We should never estimate the output of a semijoin to be more * rows than the equivalent inner join; it's obviously impossible * for that to happen. The former is N1 * Psemi while the latter * is N1 * N2 * Pinner, so we may clamp Psemi <= N2 * Pinner. * Doing this is worthwhile because of the shakier estimation * rules we use in eqjoinsel_semi, particularly in cases where it * has to punt entirely. */ selec = Min(selec, inner_rel->rows * selec_inner); That makes the funny behavior go away in both test cases shown in this thread. I find one plan change in the regression tests, but it looks reasonable enough (and checking the actual row counts shows that the estimate moved closer to reality, not further away). Now, there's a certain amount of garbage-in-garbage-out to this: if for some reason the innerjoin selectivity is way off, this could do more to hurt the semijoin estimate than to help it. But I think that generally the semijoin numbers are much less reliable than the innerjoin numbers, so mostly it ought to be no-change or a win. I'll queue this up for review in the next CF. regards, tom lane diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index f1c78ff..c00802d 100644 *** a/src/backend/utils/adt/selfuncs.c --- b/src/backend/utils/adt/selfuncs.c *************** static double ineq_histogram_selectivity *** 165,174 **** VariableStatData *vardata, FmgrInfo *opproc, bool isgt, bool iseq, Datum constval, Oid consttype); ! static double eqjoinsel_inner(Oid operator, ! VariableStatData *vardata1, VariableStatData *vardata2); ! static double eqjoinsel_semi(Oid operator, VariableStatData *vardata1, VariableStatData *vardata2, RelOptInfo *inner_rel); static bool estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, List **varinfos, double *ndistinct); --- 165,184 ---- VariableStatData *vardata, FmgrInfo *opproc, bool isgt, bool iseq, Datum constval, Oid consttype); ! static double eqjoinsel_inner(Oid opfuncoid, ! VariableStatData *vardata1, VariableStatData *vardata2, ! double nd1, double nd2, ! bool isdefault1, bool isdefault2, ! AttStatsSlot *sslot1, AttStatsSlot *sslot2, ! Form_pg_statistic stats1, Form_pg_statistic stats2, ! bool have_mcvs1, bool have_mcvs2); ! static double eqjoinsel_semi(Oid opfuncoid, VariableStatData *vardata1, VariableStatData *vardata2, + double nd1, double nd2, + bool isdefault1, bool isdefault2, + AttStatsSlot *sslot1, AttStatsSlot *sslot2, + Form_pg_statistic stats1, Form_pg_statistic stats2, + bool have_mcvs1, bool have_mcvs2, RelOptInfo *inner_rel); static bool estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, List **varinfos, double *ndistinct); *************** eqjoinsel(PG_FUNCTION_ARGS) *** 2291,2310 **** #endif SpecialJoinInfo *sjinfo = (SpecialJoinInfo *) PG_GETARG_POINTER(4); double selec; VariableStatData vardata1; VariableStatData vardata2; bool join_is_reversed; RelOptInfo *inner_rel; get_join_variables(root, args, sjinfo, &vardata1, &vardata2, &join_is_reversed); switch (sjinfo->jointype) { case JOIN_INNER: case JOIN_LEFT: case JOIN_FULL: ! selec = eqjoinsel_inner(operator, &vardata1, &vardata2); break; case JOIN_SEMI: case JOIN_ANTI: --- 2301,2369 ---- #endif SpecialJoinInfo *sjinfo = (SpecialJoinInfo *) PG_GETARG_POINTER(4); double selec; + double selec_inner; VariableStatData vardata1; VariableStatData vardata2; + double nd1; + double nd2; + bool isdefault1; + bool isdefault2; + Oid opfuncoid; + AttStatsSlot sslot1; + AttStatsSlot sslot2; + Form_pg_statistic stats1 = NULL; + Form_pg_statistic stats2 = NULL; + bool have_mcvs1 = false; + bool have_mcvs2 = false; bool join_is_reversed; RelOptInfo *inner_rel; get_join_variables(root, args, sjinfo, &vardata1, &vardata2, &join_is_reversed); + nd1 = get_variable_numdistinct(&vardata1, &isdefault1); + nd2 = get_variable_numdistinct(&vardata2, &isdefault2); + + opfuncoid = get_opcode(operator); + + memset(&sslot1, 0, sizeof(sslot1)); + memset(&sslot2, 0, sizeof(sslot2)); + + if (HeapTupleIsValid(vardata1.statsTuple)) + { + /* note we allow use of nullfrac regardless of security check */ + stats1 = (Form_pg_statistic) GETSTRUCT(vardata1.statsTuple); + if (statistic_proc_security_check(&vardata1, opfuncoid)) + have_mcvs1 = get_attstatsslot(&sslot1, vardata1.statsTuple, + STATISTIC_KIND_MCV, InvalidOid, + ATTSTATSSLOT_VALUES | ATTSTATSSLOT_NUMBERS); + } + + if (HeapTupleIsValid(vardata2.statsTuple)) + { + /* note we allow use of nullfrac regardless of security check */ + stats2 = (Form_pg_statistic) GETSTRUCT(vardata2.statsTuple); + if (statistic_proc_security_check(&vardata2, opfuncoid)) + have_mcvs2 = get_attstatsslot(&sslot2, vardata2.statsTuple, + STATISTIC_KIND_MCV, InvalidOid, + ATTSTATSSLOT_VALUES | ATTSTATSSLOT_NUMBERS); + } + + /* We need to compute the inner-join selectivity in all cases */ + selec_inner = eqjoinsel_inner(opfuncoid, + &vardata1, &vardata2, + nd1, nd2, + isdefault1, isdefault2, + &sslot1, &sslot2, + stats1, stats2, + have_mcvs1, have_mcvs2); + switch (sjinfo->jointype) { case JOIN_INNER: case JOIN_LEFT: case JOIN_FULL: ! selec = selec_inner; break; case JOIN_SEMI: case JOIN_ANTI: *************** eqjoinsel(PG_FUNCTION_ARGS) *** 2318,2329 **** inner_rel = find_join_input_rel(root, sjinfo->min_righthand); if (!join_is_reversed) ! selec = eqjoinsel_semi(operator, &vardata1, &vardata2, inner_rel); else ! selec = eqjoinsel_semi(get_commutator(operator), &vardata2, &vardata1, inner_rel); break; default: /* other values not expected here */ --- 2377,2415 ---- inner_rel = find_join_input_rel(root, sjinfo->min_righthand); if (!join_is_reversed) ! selec = eqjoinsel_semi(opfuncoid, ! &vardata1, &vardata2, ! nd1, nd2, ! isdefault1, isdefault2, ! &sslot1, &sslot2, ! stats1, stats2, ! have_mcvs1, have_mcvs2, inner_rel); else ! { ! Oid commop = get_commutator(operator); ! Oid commopfuncoid = OidIsValid(commop) ? get_opcode(commop) : InvalidOid; ! ! selec = eqjoinsel_semi(commopfuncoid, &vardata2, &vardata1, + nd2, nd1, + isdefault2, isdefault1, + &sslot2, &sslot1, + stats2, stats1, + have_mcvs2, have_mcvs1, inner_rel); + } + + /* + * We should never estimate the output of a semijoin to be more + * rows than the equivalent inner join; it's obviously impossible + * for that to happen. The former is N1 * Psemi while the latter + * is N1 * N2 * Pinner, so we may clamp Psemi <= N2 * Pinner. + * Doing this is worthwhile because of the shakier estimation + * rules we use in eqjoinsel_semi, particularly in cases where it + * has to punt entirely. + */ + selec = Min(selec, inner_rel->rows * selec_inner); break; default: /* other values not expected here */ *************** eqjoinsel(PG_FUNCTION_ARGS) *** 2333,2338 **** --- 2419,2427 ---- break; } + free_attstatsslot(&sslot1); + free_attstatsslot(&sslot2); + ReleaseVariableStats(vardata1); ReleaseVariableStats(vardata2); *************** eqjoinsel(PG_FUNCTION_ARGS) *** 2348,2396 **** * that it's worth trying to distinguish them here. */ static double ! eqjoinsel_inner(Oid operator, ! VariableStatData *vardata1, VariableStatData *vardata2) { double selec; - double nd1; - double nd2; - bool isdefault1; - bool isdefault2; - Oid opfuncoid; - Form_pg_statistic stats1 = NULL; - Form_pg_statistic stats2 = NULL; - bool have_mcvs1 = false; - bool have_mcvs2 = false; - AttStatsSlot sslot1; - AttStatsSlot sslot2; - - nd1 = get_variable_numdistinct(vardata1, &isdefault1); - nd2 = get_variable_numdistinct(vardata2, &isdefault2); - - opfuncoid = get_opcode(operator); - - memset(&sslot1, 0, sizeof(sslot1)); - memset(&sslot2, 0, sizeof(sslot2)); - - if (HeapTupleIsValid(vardata1->statsTuple)) - { - /* note we allow use of nullfrac regardless of security check */ - stats1 = (Form_pg_statistic) GETSTRUCT(vardata1->statsTuple); - if (statistic_proc_security_check(vardata1, opfuncoid)) - have_mcvs1 = get_attstatsslot(&sslot1, vardata1->statsTuple, - STATISTIC_KIND_MCV, InvalidOid, - ATTSTATSSLOT_VALUES | ATTSTATSSLOT_NUMBERS); - } - - if (HeapTupleIsValid(vardata2->statsTuple)) - { - /* note we allow use of nullfrac regardless of security check */ - stats2 = (Form_pg_statistic) GETSTRUCT(vardata2->statsTuple); - if (statistic_proc_security_check(vardata2, opfuncoid)) - have_mcvs2 = get_attstatsslot(&sslot2, vardata2->statsTuple, - STATISTIC_KIND_MCV, InvalidOid, - ATTSTATSSLOT_VALUES | ATTSTATSSLOT_NUMBERS); - } if (have_mcvs1 && have_mcvs2) { --- 2437,2451 ---- * that it's worth trying to distinguish them here. */ static double ! eqjoinsel_inner(Oid opfuncoid, ! VariableStatData *vardata1, VariableStatData *vardata2, ! double nd1, double nd2, ! bool isdefault1, bool isdefault2, ! AttStatsSlot *sslot1, AttStatsSlot *sslot2, ! Form_pg_statistic stats1, Form_pg_statistic stats2, ! bool have_mcvs1, bool have_mcvs2) { double selec; if (have_mcvs1 && have_mcvs2) { *************** eqjoinsel_inner(Oid operator, *** 2424,2431 **** nmatches; fmgr_info(opfuncoid, &eqproc); ! hasmatch1 = (bool *) palloc0(sslot1.nvalues * sizeof(bool)); ! hasmatch2 = (bool *) palloc0(sslot2.nvalues * sizeof(bool)); /* * Note we assume that each MCV will match at most one member of the --- 2479,2486 ---- nmatches; fmgr_info(opfuncoid, &eqproc); ! hasmatch1 = (bool *) palloc0(sslot1->nvalues * sizeof(bool)); ! hasmatch2 = (bool *) palloc0(sslot2->nvalues * sizeof(bool)); /* * Note we assume that each MCV will match at most one member of the *************** eqjoinsel_inner(Oid operator, *** 2435,2455 **** */ matchprodfreq = 0.0; nmatches = 0; ! for (i = 0; i < sslot1.nvalues; i++) { int j; ! for (j = 0; j < sslot2.nvalues; j++) { if (hasmatch2[j]) continue; if (DatumGetBool(FunctionCall2Coll(&eqproc, DEFAULT_COLLATION_OID, ! sslot1.values[i], ! sslot2.values[j]))) { hasmatch1[i] = hasmatch2[j] = true; ! matchprodfreq += sslot1.numbers[i] * sslot2.numbers[j]; nmatches++; break; } --- 2490,2510 ---- */ matchprodfreq = 0.0; nmatches = 0; ! for (i = 0; i < sslot1->nvalues; i++) { int j; ! for (j = 0; j < sslot2->nvalues; j++) { if (hasmatch2[j]) continue; if (DatumGetBool(FunctionCall2Coll(&eqproc, DEFAULT_COLLATION_OID, ! sslot1->values[i], ! sslot2->values[j]))) { hasmatch1[i] = hasmatch2[j] = true; ! matchprodfreq += sslot1->numbers[i] * sslot2->numbers[j]; nmatches++; break; } *************** eqjoinsel_inner(Oid operator, *** 2458,2479 **** CLAMP_PROBABILITY(matchprodfreq); /* Sum up frequencies of matched and unmatched MCVs */ matchfreq1 = unmatchfreq1 = 0.0; ! for (i = 0; i < sslot1.nvalues; i++) { if (hasmatch1[i]) ! matchfreq1 += sslot1.numbers[i]; else ! unmatchfreq1 += sslot1.numbers[i]; } CLAMP_PROBABILITY(matchfreq1); CLAMP_PROBABILITY(unmatchfreq1); matchfreq2 = unmatchfreq2 = 0.0; ! for (i = 0; i < sslot2.nvalues; i++) { if (hasmatch2[i]) ! matchfreq2 += sslot2.numbers[i]; else ! unmatchfreq2 += sslot2.numbers[i]; } CLAMP_PROBABILITY(matchfreq2); CLAMP_PROBABILITY(unmatchfreq2); --- 2513,2534 ---- CLAMP_PROBABILITY(matchprodfreq); /* Sum up frequencies of matched and unmatched MCVs */ matchfreq1 = unmatchfreq1 = 0.0; ! for (i = 0; i < sslot1->nvalues; i++) { if (hasmatch1[i]) ! matchfreq1 += sslot1->numbers[i]; else ! unmatchfreq1 += sslot1->numbers[i]; } CLAMP_PROBABILITY(matchfreq1); CLAMP_PROBABILITY(unmatchfreq1); matchfreq2 = unmatchfreq2 = 0.0; ! for (i = 0; i < sslot2->nvalues; i++) { if (hasmatch2[i]) ! matchfreq2 += sslot2->numbers[i]; else ! unmatchfreq2 += sslot2->numbers[i]; } CLAMP_PROBABILITY(matchfreq2); CLAMP_PROBABILITY(unmatchfreq2); *************** eqjoinsel_inner(Oid operator, *** 2498,2512 **** * MCVs plus non-MCV values. */ totalsel1 = matchprodfreq; ! if (nd2 > sslot2.nvalues) ! totalsel1 += unmatchfreq1 * otherfreq2 / (nd2 - sslot2.nvalues); if (nd2 > nmatches) totalsel1 += otherfreq1 * (otherfreq2 + unmatchfreq2) / (nd2 - nmatches); /* Same estimate from the point of view of relation 2. */ totalsel2 = matchprodfreq; ! if (nd1 > sslot1.nvalues) ! totalsel2 += unmatchfreq2 * otherfreq1 / (nd1 - sslot1.nvalues); if (nd1 > nmatches) totalsel2 += otherfreq2 * (otherfreq1 + unmatchfreq1) / (nd1 - nmatches); --- 2553,2567 ---- * MCVs plus non-MCV values. */ totalsel1 = matchprodfreq; ! if (nd2 > sslot2->nvalues) ! totalsel1 += unmatchfreq1 * otherfreq2 / (nd2 - sslot2->nvalues); if (nd2 > nmatches) totalsel1 += otherfreq1 * (otherfreq2 + unmatchfreq2) / (nd2 - nmatches); /* Same estimate from the point of view of relation 2. */ totalsel2 = matchprodfreq; ! if (nd1 > sslot1->nvalues) ! totalsel2 += unmatchfreq2 * otherfreq1 / (nd1 - sslot1->nvalues); if (nd1 > nmatches) totalsel2 += otherfreq2 * (otherfreq1 + unmatchfreq1) / (nd1 - nmatches); *************** eqjoinsel_inner(Oid operator, *** 2551,2559 **** selec /= nd2; } - free_attstatsslot(&sslot1); - free_attstatsslot(&sslot2); - return selec; } --- 2606,2611 ---- *************** eqjoinsel_inner(Oid operator, *** 2562,2593 **** * * (Also used for anti join, which we are supposed to estimate the same way.) * Caller has ensured that vardata1 is the LHS variable. ! * Unlike eqjoinsel_inner, we have to cope with operator being InvalidOid. */ static double ! eqjoinsel_semi(Oid operator, VariableStatData *vardata1, VariableStatData *vardata2, RelOptInfo *inner_rel) { double selec; - double nd1; - double nd2; - bool isdefault1; - bool isdefault2; - Oid opfuncoid; - Form_pg_statistic stats1 = NULL; - bool have_mcvs1 = false; - bool have_mcvs2 = false; - AttStatsSlot sslot1; - AttStatsSlot sslot2; - - nd1 = get_variable_numdistinct(vardata1, &isdefault1); - nd2 = get_variable_numdistinct(vardata2, &isdefault2); - - opfuncoid = OidIsValid(operator) ? get_opcode(operator) : InvalidOid; - - memset(&sslot1, 0, sizeof(sslot1)); - memset(&sslot2, 0, sizeof(sslot2)); /* * We clamp nd2 to be not more than what we estimate the inner relation's --- 2614,2632 ---- * * (Also used for anti join, which we are supposed to estimate the same way.) * Caller has ensured that vardata1 is the LHS variable. ! * Unlike eqjoinsel_inner, we have to cope with opfuncoid being InvalidOid. */ static double ! eqjoinsel_semi(Oid opfuncoid, VariableStatData *vardata1, VariableStatData *vardata2, + double nd1, double nd2, + bool isdefault1, bool isdefault2, + AttStatsSlot *sslot1, AttStatsSlot *sslot2, + Form_pg_statistic stats1, Form_pg_statistic stats2, + bool have_mcvs1, bool have_mcvs2, RelOptInfo *inner_rel) { double selec; /* * We clamp nd2 to be not more than what we estimate the inner relation's *************** eqjoinsel_semi(Oid operator, *** 2622,2647 **** isdefault2 = false; } ! if (HeapTupleIsValid(vardata1->statsTuple)) ! { ! /* note we allow use of nullfrac regardless of security check */ ! stats1 = (Form_pg_statistic) GETSTRUCT(vardata1->statsTuple); ! if (statistic_proc_security_check(vardata1, opfuncoid)) ! have_mcvs1 = get_attstatsslot(&sslot1, vardata1->statsTuple, ! STATISTIC_KIND_MCV, InvalidOid, ! ATTSTATSSLOT_VALUES | ATTSTATSSLOT_NUMBERS); ! } ! ! if (HeapTupleIsValid(vardata2->statsTuple) && ! statistic_proc_security_check(vardata2, opfuncoid)) ! { ! have_mcvs2 = get_attstatsslot(&sslot2, vardata2->statsTuple, ! STATISTIC_KIND_MCV, InvalidOid, ! ATTSTATSSLOT_VALUES); ! /* note: currently don't need stanumbers from RHS */ ! } ! ! if (have_mcvs1 && have_mcvs2 && OidIsValid(operator)) { /* * We have most-common-value lists for both relations. Run through --- 2661,2667 ---- isdefault2 = false; } ! if (have_mcvs1 && have_mcvs2 && OidIsValid(opfuncoid)) { /* * We have most-common-value lists for both relations. Run through *************** eqjoinsel_semi(Oid operator, *** 2664,2678 **** /* * The clamping above could have resulted in nd2 being less than ! * sslot2.nvalues; in which case, we assume that precisely the nd2 * most common values in the relation will appear in the join input, * and so compare to only the first nd2 members of the MCV list. Of * course this is frequently wrong, but it's the best bet we can make. */ ! clamped_nvalues2 = Min(sslot2.nvalues, nd2); fmgr_info(opfuncoid, &eqproc); ! hasmatch1 = (bool *) palloc0(sslot1.nvalues * sizeof(bool)); hasmatch2 = (bool *) palloc0(clamped_nvalues2 * sizeof(bool)); /* --- 2684,2698 ---- /* * The clamping above could have resulted in nd2 being less than ! * sslot2->nvalues; in which case, we assume that precisely the nd2 * most common values in the relation will appear in the join input, * and so compare to only the first nd2 members of the MCV list. Of * course this is frequently wrong, but it's the best bet we can make. */ ! clamped_nvalues2 = Min(sslot2->nvalues, nd2); fmgr_info(opfuncoid, &eqproc); ! hasmatch1 = (bool *) palloc0(sslot1->nvalues * sizeof(bool)); hasmatch2 = (bool *) palloc0(clamped_nvalues2 * sizeof(bool)); /* *************** eqjoinsel_semi(Oid operator, *** 2682,2688 **** * and because the math wouldn't add up... */ nmatches = 0; ! for (i = 0; i < sslot1.nvalues; i++) { int j; --- 2702,2708 ---- * and because the math wouldn't add up... */ nmatches = 0; ! for (i = 0; i < sslot1->nvalues; i++) { int j; *************** eqjoinsel_semi(Oid operator, *** 2692,2699 **** continue; if (DatumGetBool(FunctionCall2Coll(&eqproc, DEFAULT_COLLATION_OID, ! sslot1.values[i], ! sslot2.values[j]))) { hasmatch1[i] = hasmatch2[j] = true; nmatches++; --- 2712,2719 ---- continue; if (DatumGetBool(FunctionCall2Coll(&eqproc, DEFAULT_COLLATION_OID, ! sslot1->values[i], ! sslot2->values[j]))) { hasmatch1[i] = hasmatch2[j] = true; nmatches++; *************** eqjoinsel_semi(Oid operator, *** 2703,2712 **** } /* Sum up frequencies of matched MCVs */ matchfreq1 = 0.0; ! for (i = 0; i < sslot1.nvalues; i++) { if (hasmatch1[i]) ! matchfreq1 += sslot1.numbers[i]; } CLAMP_PROBABILITY(matchfreq1); pfree(hasmatch1); --- 2723,2732 ---- } /* Sum up frequencies of matched MCVs */ matchfreq1 = 0.0; ! for (i = 0; i < sslot1->nvalues; i++) { if (hasmatch1[i]) ! matchfreq1 += sslot1->numbers[i]; } CLAMP_PROBABILITY(matchfreq1); pfree(hasmatch1); *************** eqjoinsel_semi(Oid operator, *** 2761,2769 **** selec = 0.5 * (1.0 - nullfrac1); } - free_attstatsslot(&sslot1); - free_attstatsslot(&sslot2); - return selec; } --- 2781,2786 ---- diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out index 3ba3aaf..c55de5d 100644 *** a/src/test/regress/expected/partition_join.out --- b/src/test/regress/expected/partition_join.out *************** SELECT t1.* FROM prt1 t1 WHERE t1.a IN ( *** 801,808 **** EXPLAIN (COSTS OFF) SELECT t1.* FROM prt1 t1 WHERE t1.a IN (SELECT t1.b FROM prt2 t1 WHERE t1.b IN (SELECT (t1.a + t1.b)/2 FROM prt1_e t1 WHEREt1.c = 0)) AND t1.b = 0 ORDER BY t1.a; ! QUERY PLAN ! ------------------------------------------------------------------------------- Sort Sort Key: t1.a -> Append --- 801,808 ---- EXPLAIN (COSTS OFF) SELECT t1.* FROM prt1 t1 WHERE t1.a IN (SELECT t1.b FROM prt2 t1 WHERE t1.b IN (SELECT (t1.a + t1.b)/2 FROM prt1_e t1 WHEREt1.c = 0)) AND t1.b = 0 ORDER BY t1.a; ! QUERY PLAN ! ------------------------------------------------------------------------- Sort Sort Key: t1.a -> Append *************** SELECT t1.* FROM prt1 t1 WHERE t1.a IN ( *** 831,849 **** Index Cond: (a = t1_4.b) Filter: (b = 0) -> Nested Loop ! -> Unique ! -> Sort ! Sort Key: t1_5.b ! -> Hash Semi Join ! Hash Cond: (t1_5.b = ((t1_8.a + t1_8.b) / 2)) ! -> Seq Scan on prt2_p3 t1_5 ! -> Hash ! -> Seq Scan on prt1_e_p3 t1_8 ! Filter: (c = 0) -> Index Scan using iprt1_p3_a on prt1_p3 t1_2 Index Cond: (a = t1_5.b) Filter: (b = 0) ! (40 rows) SELECT t1.* FROM prt1 t1 WHERE t1.a IN (SELECT t1.b FROM prt2 t1 WHERE t1.b IN (SELECT (t1.a + t1.b)/2 FROM prt1_e t1 WHEREt1.c = 0)) AND t1.b = 0 ORDER BY t1.a; a | b | c --- 831,848 ---- Index Cond: (a = t1_4.b) Filter: (b = 0) -> Nested Loop ! -> HashAggregate ! Group Key: t1_5.b ! -> Hash Semi Join ! Hash Cond: (t1_5.b = ((t1_8.a + t1_8.b) / 2)) ! -> Seq Scan on prt2_p3 t1_5 ! -> Hash ! -> Seq Scan on prt1_e_p3 t1_8 ! Filter: (c = 0) -> Index Scan using iprt1_p3_a on prt1_p3 t1_2 Index Cond: (a = t1_5.b) Filter: (b = 0) ! (39 rows) SELECT t1.* FROM prt1 t1 WHERE t1.a IN (SELECT t1.b FROM prt2 t1 WHERE t1.b IN (SELECT (t1.a + t1.b)/2 FROM prt1_e t1 WHEREt1.c = 0)) AND t1.b = 0 ORDER BY t1.a; a | b | c