Why is this system swapping?

Hello,

I am trying to understand what I need to do for this system to stop using swap. Maybe it’s something simple, or obvious for the situation. I’d appreciate some thoughts/suggestions.

Some background:

This is a quad XEON (yes, Dell) with 12GB of RAM, pg 7.4…pretty heavy on concurrent usage. With peak traffic (db allows 1000 connections, in line with the number of app servers and connection pools for each) following is from ‘top’ (sorted by mem) Shared_buffers is 170MB, sort_mem 2MB. Both WAL and pgdata are on separate LUNs on fibre channel storage, RAID10.

972 processes: 971 sleeping, 1 running, 0 zombie, 0 stopped

CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle

           total   57.2%    0.0%   23.2%   0.0%     3.6%   82.8%  232.4%

           cpu00   22.0%    0.0%    9.1%   0.1%     0.9%   18.7%   48.8%

           cpu01   17.5%    0.0%    5.8%   0.0%     2.3%   19.7%   54.4%

           cpu02    7.8%    0.0%    3.7%   0.0%     0.0%   20.8%   67.5%

           cpu03    9.7%    0.0%    4.4%   0.0%     0.5%   23.6%   61.5%

Mem:  12081744k av, 12055220k used,   26524k free,       0k shrd,   71828k buff

                   9020480k actv, 1741348k in_d,  237396k in_c

Swap: 4096532k av,  472872k used, 3623660k free                 9911176k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

21397 postgres  22   0  181M 180M  175M D    25.9  1.5  85:17   0 postmaster

23820 postgres  15   0  178M 177M  175M S     0.0  1.5   1:53   3 postmaster

24428 postgres  15   0  178M 177M  175M S     0.0  1.5   1:35   3 postmaster

24392 postgres  15   0  178M 177M  175M S     2.7  1.5   2:07   2 postmaster

23610 postgres  15   0  178M 177M  175M S     0.0  1.5   0:29   2 postmaster

24395 postgres  15   0  178M 177M  175M S     0.0  1.5   1:12   1 postmaster

…

…

-bash-2.05b$ free

                  total             used       free     shared    buffers     cached

Mem:      12081744   12055536      26208          0      66704    9943988

-/+ buffers/cache:    2044844   10036900

Swap:      4096532     512744    3583788

As you can see the system starts utilizing swap at some point, with so many processes. Some time ago we had decided to keep the connections from the pool open for longer periods of time, possibly to avoid connection maintenance overhead on the db. At that time the traffic was not as high as it is today, which might be causing this, because for the most part, non-idle postmaster processes are only a few, except when the system becomes busy and suddenly you see a lot of selects piling up, and load averages shooting upwards. I am thinking closing out connections sooner might help the system release some memory to the kernel. Swapping adds up to the IO, although OS is on separate channel than postgres.

I can add more memory, but I want to make sure I haven’t missed out something obvious.

Thanks!

Anjan

******************************************************************************************

This e-mail and any files transmitted with it are intended for the use of the 

addressee(s) only and may be confidential and covered by the attorney/client 

and other privileges.  If you received this e-mail in error, please notify the 

sender; do not disclose, copy, distribute, or take any action in reliance on 

the contents of this information; and delete it from your system. Any other 

use of this e-mail is prohibited.

******************************************************************************************