Re: Limiting memory allocation

On 5/18/22 11:11, Alvaro Herrera wrote:
> On 2022-May-18, Jan Wieck wrote:
> 
>> Maybe I'm missing something, but what is it that you would actually consider
>> a solution? Knowing your current memory consumption doesn't make the need
>> for allocating some right now go away. What do you envision the response of
>> PostgreSQL to be if we had that information about resource pressure?
> 
> What was mentioned in the talk where this issue was presented, is that
> people would like malloc() to return NULL when there's memory pressure,
> even if Linux has been configured indicating that memory overcommit is
> OK.  The reason they can't set overcommit off is that it prevents other
> services in the same system from running properly.

Thank you Alvaro, that was the missing piece. Now I understand what we 
are trying to do.

> As I understand, setrlimit() sets the memory limit for any single
> process.  But that isn't useful -- the limit needed is for the whole set
> of processes under postmaster.  Limiting any individual process does no
> good.
> 
> Now that's where cgroup's memory limiting features would prove useful,
> if they weren't totally braindead:
> https://www.kernel.org/doc/Documentation/cgroup-v2.txt
> Apparently, if the cgroup goes over the "high" limit, the processes are
> *throttled*.  Then if the group goes over the "max" limit, OOM-killer is
> invoked.
> 
> (I can't see any way to make this even more counterproductive to the
> database use case.  Making the database work more slowly doesn't fix
> anything.)
> 
> So ditch cgroups.

Agreed.

> What they (Timescale) do, is have a LD_PRELOAD library that checks
> status of memory pressure, and return NULL from malloc().  This then
> leads to clean abort of transactions and all is well.  There's nothing
> that Postgres needs to do different than today.
> 
> I suppose that what they would like, is a way to inquire into the memory
> pressure status at MemoryContextAlloc() time and return NULL if it is
> too high.  How exactly this would work is unclear to me; maybe one
> process keeps an eye on it in an OS-specific manner, and if it does get
> near the maximum, set a bit in shared memory that other processes can
> examine when MemoryContextAlloc is called.  It doesn't have to be
> exactly accurate; an approximation is probably okay.

Correct, it doesn't have to be accurate. Something /proc based setting a 
flag in shared memory WOULD be good enough, IF MemoryContextAlloc() had 
some way of figuring out that its process is actually the right one to 
abort.

On a high transaction throughput system, having such a background 
process being the only one setting and clearing a flag in shared memory 
could prove disastrous. Let it check and set/clear the flag every second 
... the whole system would throw malloc(3) failures for a whole second 
on every session. Not the system I would like to benchmark ... although 
the result charts would look hilarious.

However, once we are under memory pressure to the point of aborting 
transactions, it may be reasonable to have MemoryContextAlloc() calls 
work through a queue and return NULL one by one until the pressure is 
low enough again.

I'll roll this problem around in my head for a little longer. There 
certainly is a way to do this a bit more intelligent.


Thanks again, Jan