out of memory

Hi

I have a dedicated database server with 16 GB of RAM.

and the oom-killer is killing my database

this is the output kernel:

postmaster invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Pid: 16667, comm: postmaster Tainted: G        W 2.6.27.6 #7
 [<c10510f9>] oom_kill_process+0x103/0x1d5
 [<c11c14cc>] security_capable+0x9/0xa
 [<c1050f0d>] badness+0x118/0x201
 [<c10513fc>] out_of_memory+0x231/0x265
 [<c1053300>] __alloc_pages_internal+0x322/0x447
 [<c106ac5e>] alloc_pages_current+0x59/0x89
 [<c1051c2c>] __get_free_pages+0xa/0x16
 [<c101f815>] copy_process+0xa2/0xeaa
 [<c1020683>] do_fork+0x66/0x2c3
 [<c1001a59>] sys_clone+0x32/0x36
 [<c100334a>] syscall_call+0x7/0xb
 =======================
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
CPU    8: hi:    0, btch:   1 usd:   0
CPU    9: hi:    0, btch:   1 usd:   0
CPU   10: hi:    0, btch:   1 usd:   0
CPU   11: hi:    0, btch:   1 usd:   0
CPU   12: hi:    0, btch:   1 usd:   0
CPU   13: hi:    0, btch:   1 usd:   0
CPU   14: hi:    0, btch:   1 usd:   0
CPU   15: hi:    0, btch:   1 usd:   0
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:  26
CPU    7: hi:  186, btch:  31 usd:  26
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  25
Node 0 HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:  30
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd: 167
CPU    6: hi:  186, btch:  31 usd:  25
CPU    7: hi:  186, btch:  31 usd:  19
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
CPU   12: hi:  186, btch:  31 usd:   0
CPU   13: hi:  186, btch:  31 usd:   0
CPU   14: hi:  186, btch:  31 usd:   0
CPU   15: hi:  186, btch:  31 usd:  23
Active:1573667 inactive:609150 dirty:46 writeback:107 unstable:0
 free:1750894 slab:31196 mapped:538832 pagetables:143359 bounce:0
Node 0 DMA free:3540kB min:64kB low:80kB high:96kB active:0kB inactive:0kB prese                                                                             nt:15836kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 873 16095 16095
Node 0 Normal free:28132kB min:3744kB low:4680kB high:5616kB active:840kB inacti                                                                             ve:892kB present:894080kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 121778 121778
Node 0 HighMem free:6971904kB min:512kB low:16836kB high:33160kB active:6293828k                                                                             B inactive:2435708kB present:15587608kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 113*4kB 11*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB                                                                              1*2048kB 0*4096kB = 3660kB
Node 0 Normal: 6429*4kB 147*8kB 9*16kB 2*32kB 2*64kB 1*128kB 0*256kB 1*512kB 1*1                                                                             024kB 0*2048kB 0*4096kB = 28892kB
Node 0 HighMem: 690211*4kB 364384*8kB 72902*16kB 3650*32kB 17*64kB 50*128kB 5*25                                                                             6kB 2*512kB 1*1024kB 0*2048kB 1*4096kB = 6974060kB
1920296 total pagecache pages
0 pages in swap cache
Swap cache stats: add 913280, delete 913280, find 207387/229468
Free swap  = 1754452kB
Total swap = 3904552kB
4390911 pages RAM
4161535 pages HighMem
268987 pages reserved
6066507 pages shared
1384540 pages non-shared
Out of memory: kill process 13279 (postmaster) score 546058 or a child
Killed process 13279 (postmaster)


This is the output of free -m before and after the kill

free -m
             total       used       free     shared    buffers     cached
Mem:         16101       9302       6799          0          3       7521
-/+ buffers/cache:       1776      14324
Swap:         3813       2099       1713


free -m
             total       used       free     shared    buffers     cached
Mem:         16101       5606      10494          0          7       5410
-/+ buffers/cache:        188      15912
Swap:         3813       2099       1713


the sysctl.conf  with
kernel.shmmax = 10737418240
vm.overcommit_memory = 2

I do not what why the oom-killer kill the postgres if there are available memory.

I would appreciate any help

Regars...

---

No olvides consultar tus correos desde la ventana principal de Messenger