Hey guys,
So, we have a pretty beefy system that runs dual X5675's with 72GB of
RAM. After our recent upgrade to 9.1, things have been... odd. I managed
to track it down to one setting:
shared_buffers = 8GB
The thing is, we currently have 850 clients connected to our database (I
know, that's bad, but the platform is not compatible with pgpool or
pgbouncer right now). So I did a pgbench test (scale = 3000) with 850
clients. The 3000 scale was enough to cross the NUMA barrier, because
that should force zone flushing if there's a problem. To cheat a little,
I preloaded all pgbench tables into memory with dd.
I'm not running pgbench to see performance. In this case, it's a load
test. And at first, the load test starts normally, and everything looks
fine. But if I check /proc/meminfo, I see this within 2-5 minutes:
MemFree: 461660 kB
Active(file): 23252240 kB
Inactive(file): 21272440 kB
However, if I change shared_buffers to 4GB, it ends up converging to this:
MemFree: 11024696 kB
Active(file): 46009064 kB
Inactive(file): 239672 kB
If you watch the contents of /proc/meminfo during the pgbench, it's
pretty clear when the transition starts. It only takes a couple minutes
of pgbench, and it just spirals out of control, marking tons of file
cache inactive. But according to the OS, that memory is being paged out
because the OS is out of genuinely free memory. But how does a 4GB bump
in shared_buffers wipe out 12GB?
It does the same thing at 6GB. 4GB is safe for hours on end, but 6GB and
8GB implode within in minutes. During this, kswapd goes crazy paging out
the cache, at the same time it's reading from disk to put them back in.
It's like I fed the kernel poison or something.
Has anybody else noticed something like this? I got this behavior with
9.1.6 on a 3.2 kernel. No amount of tweaks in /proc/sys/vm changed
anything either, so I'm not convinced it's a NUMA problem.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email