Re: Sort memory not being released

> > > For large allocations glibc tends to mmap() which does get
> > > unmapped. There's a threshold of 4KB I think. Ofcourse,
> > > thousands of allocations for a few bytes will never trigger it.
> >
> > But essentially all our allocation traffic goes through palloc,
> > which bunches small allocations together.  In typical scenarios
> > malloc will only see requests of 8K or more, so we should be in
> > good shape on this front.
>
> Ah, bad news. The threshold appears to be closer to 64-128KB, so for
> small allocations normal brk() calls will be made until the third or
> fourth expansion. This can be tuned (mallopt()) but that's probably
> not too good an idea.
[snip]
> Not entirely sure if it will help at all. Obviously memory
> fragmentation is your enemy here.

Depending on data use constraints and the malloc() routine in use
(this works with phk malloc() on FreeBSD, don't know about glibc or
Slowaris' routines) there's a cute trick that you can do help with
this scenario so that a large malloc()'ed region is at the end of the
data segment and therefore a process can be sbrk()'ed and shrink when
free() is called on the large allocated region.

*) malloc() the memory used in normal operations

*) malloc() the memory needed for sorting

*) free() the memory used in normal operations

*) Do whatever needs to be done with the region of memory allocated
 for sorting

*) free() the memory used for sorting

Because phk malloc() works through chains and regions, if the 1st
malloc is big enough to handle all malloc() requests during the sort
operations, the process's memory will remain reasonably unfragmented
as the chain once malloc()'ed for normal operations will be split up
and used to handle the requests during the sort operations.  Once the
sort ops are done and the sort mem is free()'ed, phk malloc will
sbrk(-1 * sort_mem) the process (shrinks the process space/releases
the top end of the data segment back to the OS).  If the malloc order
happens like: malloc() sort mem, malloc normal ops, you're hosed
because the normal ops mem region is at the top of the address space
and potentially persists longer than the sort region (likely what's
happening now), the proc can't be sbrk()'ed and the process remains
huge until the proc dies or until the regions at the top of the data
segment are free()'ed, collapsed into free contiguous regions at the
top of BSS, and then sbrk()'ed.

For long running servers and processes that grow quite large when
processing something, but you'd like to have a small foot print when
not processing data, this is what I have to do as a chump defrag
routine.  Works well for platforms that have a halfway decent
malloc().  Another option is to mmap() private anonymous regions,
though I haven't don this for anything huge yet as someone reported
being able to mmap() less than they were able to malloc()... something
I need to test.

Anyway, food for thought.  -sc

--
Sean Chittenden