Re: Memory settings, vm.overcommit, how to get it really safe?

On Thu, May 17, 2007 at 03:46:59PM +0200, Hannes Dorbath wrote:
> On a clone box I disabled overcommit, lowered PG's memory settings a
> bit, disabled swap, mirrored production load to it and monitored it how
> it would behave. As I more or less expected, it got into trouble after
> about 6 hours. All memory was exhausted, it was even unable to fork bash
> again. To my surprise I haven't found any evidence of OOM going active
> in the logs.

I think you are misunderstanding what overcommit does. Normally when
you're running programs and they fork(), the memory gets marked
copy-on-write. This data is only once in memory but if it is written by
one of the programs, they get they're own copy. Thus, it's memory
allocated but not actually used. Thus "overcommit".

Normally this is never a problem, but say that some unusual load
happens and every process with shared usage actually wants it' own
copy, and there's not enough memory+swap to hold it, something has to
give. Thus the OOM killer.

By disabling overcommit, all that happens is that in the above
situation, if the kernel sees it's overcomitting total memry+swap, it
returns ENOMEM instead. Thus instead of the unpredicatable OOM failure,
you get unpredicatable fork()/malloc()/exec() failure. For example you
can't start any more processes.

> I blamed this behaviour to the swap space I've taken away, and not to
> disabling overcommit. However I just enabled overcommit again and tried
> to reproduce the behaviour. I was unable to get it into trouble again,
> even with artificial high load.

The default setting under Linux with overcommit off are that the total
"allocated" pages in the system cannot exceed swap + 50% of memory. Thus
by removing the swap you severely limited the amount of memory that
could be used by programs. You need to give at least as much swap as
memory, otherwise you'll never get the most out of your machine.

> 1.) Why does it behave different when only changing overcommit? To my
> understanding it should have run out of memory in both cases, or can PG
> benefit from enabled overcommit? It's a minimal setup with PG being the
> only one using any noticeable amount of resources.

I hope I've answered you question above. Personally I don't disable
overcommit as I find OOM less irritating that not being able to login
when the machine is in trouble.

> 2.) Is it possible at all to put a cap on the memory PG uses in total
> from the OS side? kernel.shmmax, etc only limit some type of how PG
> might use memory? Of cause excluding OS/FS buffers etc.

You can make limit per process (see ulimit) and limit the number of
connections. If a postgres process runs out of memory, it aborts the
query.

> 3.) Can PG be made to use it's own temp files when it runs out of memory
> without setting memory settings so low that performance for typical load
> will be worse? I think it would be nice, if I wouldn't need s lot of
> swap, just to be safe under any load. Shouldn't that be more efficient
> than using paged out memory anyway?

Nope, letting the OS page is far more efficient than anything postgres
can do.

> Currently it seems to me that I have to sacrifice the performance of
> typical load, when disabling overcommit and / or reducing swap, as I
> have to push PG's memory settings lower to be safe.

Make lots and lots of swap. You'll probably never use it, but at least
it won't get in your way. I'd say 1-1.5 times your memory at least if
you want overcommit off.

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.