A couple months ago we upgraded the RAM on our database servers from 48GB to 64GB. Immediately afterwards the new RAM was being used for page cache, which is what we want, but that seems to have dropped off over time, and there's currently actually like 12GB of totally unused RAM.
What version of the Linux kernel are you using? We had exactly this problem when we were on 3.2. We've since moved to 3.8 and that solved this issue, along with a few others.
Debian squeeze, still on 2.6.32.
If you're having the same problem, this is not a NUMA issue or in any way related to zone_reclaim_mode. The memory page aging algorithm in pre 3.7 is simply broken, judging by the traffic on the Linux Kernel Mailing List (LKML).
Darn, really? I just learned about the "mysql swap insanity" problem and noticed that all the free memory is concentrated on one of the two nodes.
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6
node 0 size: 32768 MB
node 0 free: 9105 MB
node 1 cpus: 1 3 5 7
node 1 size: 32755 MB
node 1 free: 259 MB
$ free
total used free shared buffers cached
Mem: 66099280 56565804 9533476 0 11548 51788624
I haven't been able to get any traction on what that means yet though.
-- Kevin M. Goess Software Engineer Berkeley Electronic Press kgoess@bepress.com