CentOS 6.0 - 2.6.32-220.13.1.el6.x86_64
PostgreSQL version: 8.4.11
Primarily checked on PG 8.4.9 (same OS), problem also occurs. Few times aday I get a situation where PostgreSQL stops running for 1-2 minutes. CPU isrunning 99% in systime. IO is OK, only interrupts are extremely high (over100k up to 400K). System operates on 2 x Xeon 10 Core, 128 GB RAM, raid 10. Does anyone have any idea?
monitoring from atop
ATOP - 2012/05/18 09:26:16 ------ 10s elapsed
PRC | sys 6m29s | user 8m56s | #proc 1155 | #tslpi 820 | #tslpu 7 | #zombie 2 | #exit 2 |
CPU | sys 3995% | user 3% | irq 0% | idle 4% | wait 0% | curf 2.00GHz | curscal ?% |
CPL | avg1 83.19 | avg5 21.01 | avg15 9.11 | csw 3036 | intr 448760 | | numcpu 40 |
MEM | tot 126.2G | free 2.6G | cache 117.6G | dirty 17.6M | buff 277.1M | slab 1.7G | |
SWP | tot 0.0M | free 0.0M | | | | vmcom 41.5G | vmlim 63.1G |
PAG | scan 8333 | stall 0 | | | swin 0 | | swout 0 |
DSK | sda | busy 1% | read 9 | write 2 | MBr/s 0.01 | MBw/s 0.00 | avio 9.09 ms |
NET | transport | tcpi 127 | tcpo 99 | udpi 0 | udpo 3 | tcpao 0 | tcppo 36 |
NET | network | ipi 138 | ipo 103 | ipfrw 0 | deliv 130 | icmpi 0 | icmpo 0 |
NET | eth0 0% | pcki 137 | pcko 109 | si 10 Kbps | so 8 Kbps | erri 0 | erro 0 |
NET | eth1 0% | pcki 9 | pcko 0 | si 0 Kbps | so 0 Kbps | erri 0 | erro 0 |
NET | lo ---- | pcki 3 | pcko 3 | si 1 Kbps | so 1 Kbps | erri 0 | erro 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/34
64574 postgres - 0 9.44s 8m49s 0K 0K - - NE 0 E - 100% <postmaster>
23 root root 1 5.22s 0.00s 0K 0K 0K 0K -- - S 5 50% migration/5
26885 postgres postgres 1 4.93s 0.00s 0K 4096K 0K 0K -- - R 25 47% postmaster
71 root