Hi Alexander,
On Sun, Dec 24, 2023 at 5:32 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
>
> Thank you Ashutosh for your work on this matter. With a large number of partitions, it definitely makes sense to
reduceboth Bitmapset's size as well as the number of Bitmapsets.
>
> I've checked the patchset [1] with your test suite to check the memory consumption. The results are in the table
below.
>
> query | no patch | patch | no self-join removal
> ----------------------------------------------------------------------------------
> 2-way join, non partitioned | 14792 | 15208 | 29152
> 2-way join, no partitionwise join | 19519576 | 19519576 | 19519576
> 2-way join, partitionwise join | 40851968 | 40851968 | 40851968
> 3-way join, non partitioned | 20632 | 21784 | 79376
> 3-way join, no partitionwise join | 45227224 | 45227224 | 45227224
> 3-way join, partitionwise join | 151655144 | 151655144 | 151655144
> 4-way join, non partitioned | 25816 | 27736 | 209128
> 4-way join, no partitionwise join | 83540712 | 83540712 | 83540712
> 4-way join, partitionwise join | 463960088 | 463960088 | 463960088
> 5-way join, non partitioned | 31000 | 33720 | 562552
> 5-way join, no partitionwise join | 149284376 | 149284376 | 149284376
> 5-way join, partitionwise join | 1663896608 | 1663896608 | 1663896608
>
>
> The most noticeable thing for me is that self-join removal doesn't work with partitioned tables. I think this is the
directionfor future work on this subject. In non-partitioned cases, patchset gives a small memory overhead. However,
thememory consumption is still much less than it is without the self-join removal. So, removing the join still lowers
memoryconsumption even if it copies some Bitmapsets. Given that patchset [1] is required for the correctness of memory
manipulationsin Bitmapsets during join removals, I'm going to push it if there are no objections.
I am missing the link between this work and the self join work. Can
you please provide me relevant pointers?
--
Best Wishes,
Ashutosh Bapat