John R Allgood <jallgood@the-allgoods.net> writes:
> Here is the output from free on one of the nodes.
Hmmm ... I'm not exactly a kernel jock, but I find the presence of these
lines in the output to be mighty suspicious:
> Low: 821 510 310
> High: 7294 5459 1835
In a true 64-bit system you should not have any distinction between low
and high memory (and indeed "free" doesn't print any such thing on my
x86_64 box). Maybe you are running a 32-bit kernel? Anyway it seems
likely that the out-of-memory situation was due to oversubscribed lowmem
rather than being out of memory globally, especially since you are
running at zero swap usage. This is not the trace of a system that's
under any memory pressure overall:
> total used free
> Mem: 8116 5969 2146
> -/+ buffers/cache: 1506 6609
> Swap: 2000 0 1999
Something else to check is whether having swap only a quarter the size
of physical RAM is a good idea or not. I'm not sure what the latest
configuration theories are for Linux, but the old-school-Unix theory was
always that you should have more swap space than RAM. When memory
overcommit is disabled, having plenty of swap space available may be
necessary even if you're seemingly not using it --- the kernel needs to
be sure that there would be someplace to put everything if it had to
materialize all the virtual copy-on-write pages that the current process
set is sharing.
regards, tom lane