Обсуждение: Memory management with C aggregate

We have user-defined aggregates written in C running inside the server.  We are running into some memory management
issuesand wondering what 
 
is the best way to solve the problem.

The state of the aggregates is a structure with a pointer to allocated 
memory.  The structure and memory are allocated in the 
PortalMemoryContext and freed in the ffunc.  We just discovered that the 
ffunc function can be called multiple times with the same state on 
certain queries.  The double free causes a crash.
From what I could find, the proper way to handle this is let the memory 
be freed when the memory context is freed.  What is the right memory 
context to be used to allocate the aggregate state?
 - Ian

Ian Burrell <imb@rentrak.com> writes:
> We have user-defined aggregates written in C running inside the server. 
>   We are running into some memory management issues and wondering what 
> is the best way to solve the problem.

> The state of the aggregates is a structure with a pointer to allocated 
> memory.  The structure and memory are allocated in the 
> PortalMemoryContext and freed in the ffunc.  We just discovered that the 
> ffunc function can be called multiple times with the same state on 
> certain queries.  The double free causes a crash.

AFAICS the ffunc should be called only once per aggregated group.
Perhaps your code is confused about the handling of groups?  If so,
the double free is hardly your only worry --- you'll be computing wrong
answers anyway till you fix that.

>  From what I could find, the proper way to handle this is let the memory 
> be freed when the memory context is freed.  What is the right memory 
> context to be used to allocate the aggregate state?

I'd use the memory context identified by fcinfo->flinfo->fn_mcxt.
        regards, tom lane

Tom Lane wrote:
> Ian Burrell <imb@rentrak.com> writes:
> 
>>We have user-defined aggregates written in C running inside the server. 
>>  We are running into some memory management issues and wondering what 
>>is the best way to solve the problem.
> 
> 
>>The state of the aggregates is a structure with a pointer to allocated 
>>memory.  The structure and memory are allocated in the 
>>PortalMemoryContext and freed in the ffunc.  We just discovered that the 
>>ffunc function can be called multiple times with the same state on 
>>certain queries.  The double free causes a crash.
> 
> 
> AFAICS the ffunc should be called only once per aggregated group.
> Perhaps your code is confused about the handling of groups?  If so,
> the double free is hardly your only worry --- you'll be computing wrong
> answers anyway till you fix that.
> 

The aggregate is in a subquery which might make a difference.  The ffunc 
is only called multiple times when a nested loop is used in the query 
plan.  With other query plans, the ffunc is only called once.  The ffunc 
seems to be called once for every combination.

I can't get the following query to use a nested loop, but our query is 
similar, but with a compound key and a custom aggregate.  If x has N 
rows, y has M, with N groups, then the ffunc can be called N*M times, M 
times for each N group.

SELECT x.key_no, s.agg
FROM x
INNER JOIN (        SELECT y.key_no, custom_agg(num) AS agg        FROM y        GROUP BY key_no
) s
USING (key_no)
ORDER BY key_no

I'll try to come up with a test case that illustrates the problem.
 - Ian