Re: Spilling hashed SetOps and aggregates to disk

On 06/05/2018 03:41 PM, serge@rielau.com wrote:
> In our code base we have added a WithStats-Flavor for creating memory 
> contexts.
> This api accepts a pointer to metric for accounting and it is inherited 
> by all subcontexts unless overridden.
> So we only needed to change context creation API where we wanted (such 
> as TopTansactionContext, Message Context, ..)
> That's quite trivial, actually.

I think that's pretty much what we tried when this patch was first 
discussed in 2015, but it added measurable overhead (1-3%) even when the 
accounting was disabled. Which is not quite desirable, of course.

> Also we have fixed all those missing hash spills - albeit based on the 
> 9.6 hash table design I think.

I don't think this part of the code changed all that much.

> If there is interest by the community we are very willing to share.

Absolutely! I think it'd be great to share both the code and the 
reasoning behind the design.

> Cheers
> Serge
> Salesforce

cheers

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services