While testing the just-committed executor memory leak fixes, I observe
that there are still slow leaks in some operations such as sorts on
large text fields. With a little digging, it seems that the leak must
be blamed on the behavior of aset.c for large chunks. Chunks between
1K and 64K (ALLOC_SMALLCHUNK_LIMIT and ALLOC_BIGCHUNK_LIMIT) are all
kept in the same freelist, and when a request is made for an amount
of memory in that range, aset.c gives back the first chunk that's
big enough from that freelist.
For example, let's say you are allocating and freeing roughly equal
numbers of 2K and 10K blocks. Over time, about half of the 2K
requests will be answered by returning a 10K block --- which will
prevent the next 10K request from being filled by recycling that
10K block, causing a new 10K block to be allocated. If aset.c
had given back a 2K block whenever possible, the net memory usage
would be static, but since it doesn't, the usage gradually creeps
up as more and more chunks are used inefficiently. Our actual
maximum memory usage might be m * 2K plus n * 10K but the allocated
space will creep towards (m + n) * 10K where *all* the active blocks
are the larger size.
A straightforward solution would be to scan the entire freelist
and give back the smallest block that's big enough for the request.
That could be pretty slow (and induce lots of swap thrashing) so
I don't like it much.
Another idea is to split the returned chunk and put the wasted part back
as a smaller free chunk, but I don't think this solves the problem; it
just means that the wasted space ends up on a small-chunk freelist, not
that you can actually do anything with it. But maybe someone can figure
out a variant that works better.
It might be that this behavior won't be seen much in practice and we
shouldn't slow down aset.c at all to try to deal with it. But I think
it's worth looking for solutions that won't slow typical cases much.
regards, tom lane