Memory management with C aggregate

We have user-defined aggregates written in C running inside the server.  We are running into some memory management
issuesand wondering what 
 
is the best way to solve the problem.

The state of the aggregates is a structure with a pointer to allocated 
memory.  The structure and memory are allocated in the 
PortalMemoryContext and freed in the ffunc.  We just discovered that the 
ffunc function can be called multiple times with the same state on 
certain queries.  The double free causes a crash.
From what I could find, the proper way to handle this is let the memory 
be freed when the memory context is freed.  What is the right memory 
context to be used to allocate the aggregate state?
 - Ian