kswapd 100%, swap full, vm.swappiness=0

Hardware:
48 core AMD Magny Cours (4x12)
128G 1333MHz memory
34 15k6 drives, 2 hot spares, rest in RAID-1 pairs, 1 set for OS, 4
for pg_xlog, rest for /data/base
LSI 8888 RAID controller
OS:
Ubuntu 10.04

uname -a
Linux bigassdbserver 2.6.32-24-generic #38-Ubuntu SMP Mon Jul 5
09:20:59 UTC 2010 x86_64 GNU/Linux

scheduler = noop for all drive sets.
Settings for sysctl.conf:
 vm.zone_reclaim_mode = 0
 kernel.shmmax = 33554432000
 kernel.shmall = 2097152000
 kernel.shmmni = 4096
 vm.swappiness = 0
 vm.dirty_ratio = 2
 vm.dirty_background_ratio = 1

$ free
             total       used       free     shared    buffers     cached
Mem:     131651412  104986524   26664888          0     910804   91170764
-/+ buffers/cache:   12904956  118746456
Swap:            0          0          0

(swap is now off with sudo swapoff -a, it fixed the problem)

It's twin, the read slave, looks like this:

$ free
             total       used       free     shared    buffers     cached
Mem:     131651412  110364700   21286712          0     702144   96771656
-/+ buffers/cache:   12890900  118760512
Swap:     25388024        940   25387084

So, this morning, the machine goes into 100% swap usage.  four kswapds
are running at 100% CPU in mostly D state.  Load climbs to 300.
Server gets a little slow.  Swapoff -a fixes it.

This makes no sense to me.  The machine had 90G+ in kernel cache, and
was NOT running out of memory in any way.  Swappiness is 0.

Any advice on this, reporting it to the kernel guys etc welcome.

--
To understand recursion, one must first understand recursion.