Обсуждение: Patch: Range Merge Join

Поиск
Список
Период
Сортировка

Patch: Range Merge Join

От
Thomas
Дата:
Hi Hackers,

More than a year ago we submitted a patch that offered two primitives
(ALIGN and NORMALIZE) to support the processing of temporal data with range types. During the ensuing discussion we decided to withdraw the original patch
and to split it into smaller parts.

In the context of my BSc thesis, we started working and implementing a Range Merge Join (RMJ), which is key for most temporal operations. The RMJ is a useful operator in its own right and it greatly benefits any possible temporal extension.

We have implemented the Range Merge Join algorithm by extending the
existing Merge Join to also support range conditions, i.e., BETWEEN-AND
or @> (containment for range types). Range joins contain a containment
condition and may have (optional) equality conditions. For example the
following query joins employees with a department and work period with
events on a specific day for that department:

SELECT emps.name, emps.dept, events.event, events.day
FROM emps JOIN events ON emps.dept = events.dept
AND events.day <@ emps.eperiod;

The resulting query plan is as follows:


                                          QUERY PLAN

----------------------------------------------------------------------------------------------
 Range Merge Join  (cost=106.73..118.01 rows=3 width=100) (actual rows=6
loops=1)
   Merge Cond: (emps.dept = events.dept)
   Range Cond: (events.day <@ emps.eperiod)
   ->  Sort  (cost=46.87..48.49 rows=650 width=96) (actual rows=5 loops=1)
         Sort Key: emps.dept, emps.eperiod
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on emps  (cost=0.00..16.50 rows=650 width=96) (actual
rows=5 loops=1)
   ->  Sort  (cost=59.86..61.98 rows=850 width=68) (actual rows=6 loops=1)
         Sort Key: events.dept, events.day
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on events  (cost=0.00..18.50 rows=850 width=68)
(actual rows=5 loops=1)
 Planning Time: 0.077 ms
 Execution Time: 0.092 ms
(13 rows)


Example queries and instances of tables can be found at the end of the mail.

The range merge join works with range types using <@ and also scalar data types
using "a.ts BETWEEN b.ts AND b.te" or "b.ts <= a.ts AND a.ts <= b.te".
Currently, PostgreSQL does not provide specialized join algorithms for range
conditions (besides index nested loops), or Hash Join and Merge Joins that
evaluate an equality condition only.

Our idea is to have a separate range_cond besides the merge_cond for the
Merge Join that stores the potential range conditions of a query. The state
diagram of the Merge Join is then extended to also take into consideration
the range_cond. See the simplified state diagram of the Range Merge Join as
an extension of the Merge Join in the attachment. These additions besides a
boolean check have no effect on the Marge Join when no range condition is
present.

We provide extensive testing results and further information, including the
full BSc Thesis (technical report), describing the implementation and tests
in detail on http://tpg.inf.unibz.it/project-rmj and
http://tpg.inf.unibz.it/downloads/rmj-report.pdf.

We performed several experiments and show that depending on the selectivity of
the range condition the range merge join outperforms existing execution
algorithms up to an order of magnitude. We found that the range merge join that
needs to find range_cond from inequalities, incurs only a very small overhead
in planning time in some TPCH queries (see Table 5.3 in the technical report)
and in general only a very small overhead for a large number of joins or many
inequality conditions (see Figure 5.1). To check the overhead of our extension
for the traditional merge join execution time, we executed the TPCH queries
using the merge join (hash join disabled) and found no statistically
significant difference (see Table 5.4).

We are looking forward to your feedback and any suggestions to improve the patch.

Best Regards,

Thomas Mannhart


Attachments: State Diagram and Patch

OPEN POINTS AND TODOs:

- Currently we do not consider parallelization
- Not all cases for input sort orders are considered yet

EXAMPLE QUERIES:

The first query uses a range condition using BETWEEN AND only and no
equality condition.

----------------------------------------------------------------------------------------------

DROP TABLE IF EXISTS marks;
DROP TABLE IF EXISTS grades;

CREATE TABLE marks (name text, snumber numeric, mark numeric);
CREATE TABLE grades (mmin numeric, mmax numeric, grade numeric);

INSERT INTO marks (name, snumber, mark) VALUES
('Anton', 1232, 23.5),
('Thomas', 4356, 95),
('Michael', 1125, 72),
('Hans', 3425, 90);

INSERT INTO grades (mmin, mmax, grade) VALUES
(0.0, 18, 1),
(18.5, 36, 2),
(36.5, 54, 3),
(54.5, 72, 4),
(72.5, 90, 5),
(90.5, 100, 6);

EXPLAIN(ANALYZE, TIMING FALSE)
SELECT marks.name, marks.snumber, grades.grade
FROM marks JOIN grades ON marks.mark BETWEEN grades.mmin AND grades.mmax;

                                          QUERY PLAN

-----------------------------------------------------------------------------------------------
 Range Merge Join  (cost=93.74..920.13 rows=46944 width=96) (actual rows=16
loops=1)
   Range Cond: ((marks.mark >= grades.mmin) AND (marks.mark <= grades.mmax))
   ->  Sort  (cost=46.87..48.49 rows=650 width=96) (actual rows=12 loops=1)
         Sort Key: grades.mmin
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on grades  (cost=0.00..16.50 rows=650 width=96)
(actual rows=12 loops=1)
   ->  Sort  (cost=46.87..48.49 rows=650 width=96) (actual rows=21 loops=1)
         Sort Key: marks.mark
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on marks  (cost=0.00..16.50 rows=650 width=96)
(actual rows=8 loops=1)
 Planning Time: 0.078 ms
 Execution Time: 0.068 ms
(12 rows)

----------------------------------------------------------------------------------------------


The second query uses a range and an equality condition and joins the
relations using contained in (<@).

----------------------------------------------------------------------------------------------

DROP TABLE IF EXISTS emps;
DROP TABLE IF EXISTS events;

CREATE TABLE emps (name text, dept text, eperiod daterange);
CREATE TABLE events (event text, dept text, day date);

INSERT INTO emps (name, dept, eperiod) VALUES
('Anton', 'Sales', '(2020-01-01, 2020-03-31)'),
('Thomas', 'Marketing', '(2020-01-01, 2020-06-30)'),
('Michael', 'Marketing', '(2020-03-01, 2020-12-31)'),
('Hans', 'Sales', '(2020-01-01, 2020-12-31)'),
('Thomas', 'Accounting', '(2020-07-01, 2020-12-31)');

INSERT INTO events (event, dept, day) VALUES
('Fair CH', 'Marketing', '2020-03-05'),
('Presentation', 'Sales', '2020-06-15'),
('Fair IT', 'Marketing', '2020-08-03'),
('Balance Report', 'Accounting', '2020-08-03'),
('Product launch', 'Marketing', '2020-10-15');

EXPLAIN(ANALYZE, TIMING FALSE)
SELECT emps.name, emps.dept, events.event, events.day
FROM emps JOIN events ON emps.dept = events.dept
AND events.day <@ emps.eperiod;

                                          QUERY PLAN

----------------------------------------------------------------------------------------------
 Range Merge Join  (cost=106.73..118.01 rows=3 width=100) (actual rows=6
loops=1)
   Merge Cond: (emps.dept = events.dept)
   Range Cond: (events.day <@ emps.eperiod)
   ->  Sort  (cost=46.87..48.49 rows=650 width=96) (actual rows=5 loops=1)
         Sort Key: emps.dept, emps.eperiod
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on emps  (cost=0.00..16.50 rows=650 width=96) (actual
rows=5 loops=1)
   ->  Sort  (cost=59.86..61.98 rows=850 width=68) (actual rows=6 loops=1)
         Sort Key: events.dept, events.day
         Sort Method: quicksort  Memory: 25kB
         ->  Seq Scan on events  (cost=0.00..18.50 rows=850 width=68)
(actual rows=5 loops=1)
 Planning Time: 0.077 ms
 Execution Time: 0.092 ms
(13 rows)

----------------------------------------------------------------------------------------------
Вложения

Re: Patch: Range Merge Join

От
David Rowley
Дата:
On Thu, 10 Jun 2021 at 03:05, Thomas <thomasmannhart97@gmail.com> wrote:
> We have implemented the Range Merge Join algorithm by extending the
> existing Merge Join to also support range conditions, i.e., BETWEEN-AND
> or @> (containment for range types).

It shouldn't be a blocker for you, but just so you're aware, there was
a previous proposal for this in [1] and a patch in [2].  I've include
Jeff here just so he's aware of this. Jeff may wish to state his
intentions with his own patch. It's been a few years now.

I only just glanced over the patch. I'd suggest getting rid of the /*
Thomas */ comments.  We use git, so if you need an audit trail about
changes then you'll find it in git blame.  If you have those for an
internal audit trail then you should consider using git. No committer
would commit those to PostgreSQL, so they might as well disappear.

For further review, please add the patch to the July commitfest [3].
We should be branching for pg15 sometime before the start of July.
There will be more focus on new patches around that time. Further
details in [4].

Also, I see this if your first post to this list, so welcome, and
thank you for the contribution.  Also, just to set expectations;
patches like this almost always take a while to get into shape for
PostgreSQL. Please expect a lot of requests to change things. That's
fairly standard procedure.  The process often drags on for months and
in some less common cases, years.

David

[1] https://www.postgresql.org/message-id/flat/6227.1334559170%40sss.pgh.pa.us#82c771950ba486dec911923a5e910000
[2] https://www.postgresql.org/message-id/flat/CAMp0ubfwAFFW3O_NgKqpRPmm56M4weTEXjprb2gP_NrDaEC4Eg%40mail.gmail.com
[3] https://commitfest.postgresql.org/33/
[4] https://wiki.postgresql.org/wiki/CommitFest



Re: Patch: Range Merge Join

От
Thomas
Дата:
Thank you for the feedback.
I removed the redundant comments from the patch and added this thread to the July CF [1].

Best Regards,
Thomas Mannhart


Am Do., 10. Juni 2021 um 05:10 Uhr schrieb David Rowley <dgrowleyml@gmail.com>:
On Thu, 10 Jun 2021 at 03:05, Thomas <thomasmannhart97@gmail.com> wrote:
> We have implemented the Range Merge Join algorithm by extending the
> existing Merge Join to also support range conditions, i.e., BETWEEN-AND
> or @> (containment for range types).

It shouldn't be a blocker for you, but just so you're aware, there was
a previous proposal for this in [1] and a patch in [2].  I've include
Jeff here just so he's aware of this. Jeff may wish to state his
intentions with his own patch. It's been a few years now.

I only just glanced over the patch. I'd suggest getting rid of the /*
Thomas */ comments.  We use git, so if you need an audit trail about
changes then you'll find it in git blame.  If you have those for an
internal audit trail then you should consider using git. No committer
would commit those to PostgreSQL, so they might as well disappear.

For further review, please add the patch to the July commitfest [3].
We should be branching for pg15 sometime before the start of July.
There will be more focus on new patches around that time. Further
details in [4].

Also, I see this if your first post to this list, so welcome, and
thank you for the contribution.  Also, just to set expectations;
patches like this almost always take a while to get into shape for
PostgreSQL. Please expect a lot of requests to change things. That's
fairly standard procedure.  The process often drags on for months and
in some less common cases, years.

David

[1] https://www.postgresql.org/message-id/flat/6227.1334559170%40sss.pgh.pa.us#82c771950ba486dec911923a5e910000
[2] https://www.postgresql.org/message-id/flat/CAMp0ubfwAFFW3O_NgKqpRPmm56M4weTEXjprb2gP_NrDaEC4Eg%40mail.gmail.com
[3] https://commitfest.postgresql.org/33/
[4] https://wiki.postgresql.org/wiki/CommitFest
Вложения

Re: Patch: Range Merge Join

От
Jeff Davis
Дата:
On Thu, 2021-06-10 at 15:09 +1200, David Rowley wrote:
> It shouldn't be a blocker for you, but just so you're aware, there
> was
> a previous proposal for this in [1] and a patch in [2].  I've include
> Jeff here just so he's aware of this. Jeff may wish to state his
> intentions with his own patch. It's been a few years now.

Great, thank you for working on this!

I'll start with the reason I set the work down before: it did not work
well with multiple join keys. That might be fine, but I also started
thinking it was specialized enough that I wanted to look into doing it
as an extension using the CustomScan mechanism.

Do you have any solution to working better with multiple join keys? And
do you have thoughts on whether it would be a good candidate for the
CustomScan extension mechanism, which would make it easier to
experiment with?

Regards,
    Jeff Davis





Re: Patch: Range Merge Join

От
Jaime Casanova
Дата:
On Thu, Jun 10, 2021 at 07:14:32PM -0700, Jeff Davis wrote:
> On Thu, 2021-06-10 at 15:09 +1200, David Rowley wrote:
> > It shouldn't be a blocker for you, but just so you're aware, there
> > was
> > a previous proposal for this in [1] and a patch in [2].  I've include
> > Jeff here just so he's aware of this. Jeff may wish to state his
> > intentions with his own patch. It's been a few years now.
> 
> Great, thank you for working on this!
> 
> I'll start with the reason I set the work down before: it did not work
> well with multiple join keys. That might be fine, but I also started
> thinking it was specialized enough that I wanted to look into doing it
> as an extension using the CustomScan mechanism.
> 
> Do you have any solution to working better with multiple join keys? And
> do you have thoughts on whether it would be a good candidate for the
> CustomScan extension mechanism, which would make it easier to
> experiment with?
> 

Hi,

It seems this has been stalled since jun-2021. I intend mark this as
RwF unless someone speaks in the next hour or so.

-- 
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL



Re: Patch: Range Merge Join

От
Jaime Casanova
Дата:
> On Mon, Oct 04, 2021 at 04:27:54PM -0500, Jaime Casanova wrote:
>> On Thu, Jun 10, 2021 at 07:14:32PM -0700, Jeff Davis wrote:
>> > 
>> > I'll start with the reason I set the work down before: it did not work
>> > well with multiple join keys. That might be fine, but I also started
>> > thinking it was specialized enough that I wanted to look into doing it
>> > as an extension using the CustomScan mechanism.
>> > 
>> > Do you have any solution to working better with multiple join keys? And
>> > do you have thoughts on whether it would be a good candidate for the
>> > CustomScan extension mechanism, which would make it easier to
>> > experiment with?
>> > 
>> 
>> Hi,
>> 
>> It seems this has been stalled since jun-2021. I intend mark this as
>> RwF unless someone speaks in the next hour or so.
>> 

Thomas <thomasmannhart97@gmail.com> wrote me:

> Hi,
> 
> I registered this patch for the commitfest in july. It had not been reviewed and moved to the next CF. I still like
tosubmit it.
 
> 
> Regards,
> Thomas
>

Just for clarification RwF doesn't imply reject of the patch.
Nevertheless, given that there has been no real review I will mark this
patch as "Waiting on Author" and move it to the next CF.

Meanwhile, cfbot (aka http://commitfest.cputube.org) says this doesn't
compile. Here is a little patch to fix the compilation errors, after
that it passes all tests in make check-world.

Also attached a rebased version of your patch with the fixes so we turn
cfbot entry green again

-- 
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL

Вложения

Re: Patch: Range Merge Join

От
Daniel Gustafsson
Дата:
This patch fails to compile due to an incorrect function name in an assertion:

  nodeMergejoin.c:297:9: warning: implicit declaration of function 'list_legth' is invalid in C99
[-Wimplicit-function-declaration]
  Assert(list_legth(node->rangeclause) < 3);
         ^

--
Daniel Gustafsson        https://vmware.com/




Re: Patch: Range Merge Join

От
Tomas Vondra
Дата:
On 11/17/21 15:45, Thomas wrote:
> Thank you for the feedback and sorry for the oversight. I fixed the bug 
> and attached a new version of the patch.
> 
> Kind Regards, Thomas
> 
> Am Mi., 17. Nov. 2021 um 15:03 Uhr schrieb Daniel Gustafsson 
> <daniel@yesql.se <mailto:daniel@yesql.se>>:
> 
>     This patch fails to compile due to an incorrect function name in an
>     assertion:
> 
>        nodeMergejoin.c:297:9: warning: implicit declaration of function
>     'list_legth' is invalid in C99 [-Wimplicit-function-declaration]
>        Assert(list_legth(node->rangeclause) < 3);
>

That still doesn't compile with asserts, because MJCreateRangeData has

     Assert(list_length(node->rangeclause) < 3);

but there's no 'node' variable :-/


I took a brief look at the patch, and I think there are two main issues 
preventing it from moving forward.

1) no tests

There's not a *single* regression test exercising the new code, so even 
after adding Assert(false) to MJCreateRangeData() tests pass just fine. 
Clearly, that needs to change.

2) lack of comments

The patch adds a bunch of functions, but it does not really explain what 
the functions do (unlike the various surrounding functions). Even if I 
can work out what the functions do, it's much harder to determine what 
the "contract" is (i.e. what assumptions the function do and what is 
guaranteed).

Similarly, the patch modifies/reworks large blocks of executor code, 
without updating the comments describing what the block does.

See 0002 for various places that I think are missing comments.


Aside from that, I have a couple minor comments:

3) I'm not quite sure I like "Range Merge Join" to be honest. It's still 
a "Merge Join" pretty much. What about ditching the "Range"? There'll 
still be "Range Cond" key, which should be good enough I think.

4) Some minor whitespace issues (tabs vs. spaces). See 0002.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Вложения

Re: Patch: Range Merge Join

От
Julien Rouhaud
Дата:
Hi,

On Wed, Nov 17, 2021 at 11:28:43PM +0100, Tomas Vondra wrote:
> On 11/17/21 15:45, Thomas wrote:
> > Thank you for the feedback and sorry for the oversight. I fixed the bug
> > and attached a new version of the patch.
> > 
> > Kind Regards, Thomas
> > 
> > Am Mi., 17. Nov. 2021 um 15:03 Uhr schrieb Daniel Gustafsson
> > <daniel@yesql.se <mailto:daniel@yesql.se>>:
> > 
> >     This patch fails to compile due to an incorrect function name in an
> >     assertion:
> > 
> >        nodeMergejoin.c:297:9: warning: implicit declaration of function
> >     'list_legth' is invalid in C99 [-Wimplicit-function-declaration]
> >        Assert(list_legth(node->rangeclause) < 3);
> > 
> 
> That still doesn't compile with asserts, because MJCreateRangeData has
> 
>     Assert(list_length(node->rangeclause) < 3);
> 
> but there's no 'node' variable :-/
> 
> 
> I took a brief look at the patch, and I think there are two main issues
> preventing it from moving forward.
> 
> 1) no tests
> 
> 2) lack of comments
> 
> 3) I'm not quite sure I like "Range Merge Join" to be honest. It's still a
> "Merge Join" pretty much. What about ditching the "Range"? There'll still be
> "Range Cond" key, which should be good enough I think.
> 
> 4) Some minor whitespace issues (tabs vs. spaces). See 0002.

It's been 2 months since Tomas posted that review.

Thomas, do you plan to work on that patch during this commitfest?