On 07/19/2016 06:22 PM, Tomas Vondra wrote:
>
> [one huge cup of coffee later ...]
>
> So, I think it's actually quite simple to improve this by using a
> generational design, i.e. creating a sequence of slab contexts every N
> tuples, using the average tuple length from the previous batch as input
> for the next one (multiplied by 1.5x to give a bit of slack). Maybe this
> heuristics is far too simple, but perhaps we can
>
> Attached is a v2 of the patch doing pretty much this - I wouldn't call
> this even half-baked, it's a rather early PoC thingy. But it seems to
> work, and for the test case from the initial report, it significantly
> reduces memory consumption by a factor of 100x, as the average tuple
> length is ~80B, not 8kB. It increases the runtime a bit (say, from 700ms
> to 800ms for the 100k test case), but that seems negligible compared to
> the master behavior and worth considering the memory consumption reduction.
>
> This relies on one of the SlabContext features - it can easily see which
> blocks are free because it has 'free bitmap' in each block, so old
> contexts get automatically freed once the tuples are pfreed. I have not
> made any attempt to free the context structure itself, which is a memory
> leak, but I feel lazy and it's good enough for a PoC.
>
> The AllocSet obviously can't do that, so that part is not part of the
> generation design - it would just prevent reuse of the memory, thus
> increasing memory consumption. But if we could make the AllocSet capable
> of doing AllocSetIsEmpty reasonably (and not just when it's reset, which
> is pretty useless I believe), we could discard the whole context (making
> the generational design viable).
>
Heck, we could even implement this as a special type of memory context,
suitable for cases with mostly the same palloc sizes (but not entirely
constant, as required by the SlabContext as implemented in the patch).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services