The memory context stats from a running process before it gets killed by OOM look like this
TopMemoryContext: 101560 total in 6 blocks; 7336 free (6 chunks); 94224 used TopTransactionContext: 73816 total in 4 blocks; 11624 free (0 chunks); 62192 used ExecutorState: 1375731712 total in 174 blocks; 5391392 free (382 chunks); 1370340320 used HashAgg meta context: 315784 total in 10 blocks; 15400 free (2 chunks); 300384 used ExprContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used ExprContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used ExprContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used ...
That's 1.3GB allocated in ExecutorState - that doesn't seem right.
FWIW there are only very few groups (each attribute has fewer than 30 distinct values, so there's only about ~1000 groups. On master it works just fine, of course.
Thanks a lot, the patch has a memory leak in the lookup_hash_entries, it uses a list_concat there
and causes a 64-byte leak for every tuple, has fixed that.