How to find out why user processes are using so much memory?

Hi,

Having an application on Postgres.
Most parameters are on default.
Memory related parameters are slightly increased:
effective_cache_size = 512MB
max_connections = 300
maintenance_work_mem = 64MB
shared_buffers = 512MB
wal_buffers = 16MB
work_mem = 10MB

92 user processes are using around 30G RAM + 20G Swap at the moment.
pg_top shows an even distribution of RAM per process (see below).

It seems the application is eating up more and more memory.
Any idea how to find out why the user processes are using that much memory?

Is there a statistic memory usage per statement or something like that?
pg_stat_statements is only showing shared_buffers per statement.

-Markus



last pid: 92572;  load avg:  0.22,  0.32,  0.22;       up 258+21:20:34
                           11:22:16 
96 processes: 96 sleeping
CPU states:  0.2% user,  0.0% nice,  0.2% system, 99.6% idle,  0.0% iowait
Memory: 45G used, 2091M free, 624K buffers, 13G cached
DB activity:  33 tps,  0 rollbs/s,  81 buffer r/s, 98 hit%,    479 row r/s,    7 row w/s s
DB I/O:     5 reads/s,   171 KB/s,     2 writes/s,    19 KB/s
DB disk: 190.0 GB total, 28.2 GB free (85% used)
Swap: 19G used, 3407M free, 90M cached

  PID USERNAME PRI NICE  SIZE   RES STATE   TIME   WCPU    CPU COMMAND
48064 postgres  20    0  788M  146M sleep   0:03  0.02%  0.20% postgres: pcl_p011: pdb_ppp_oiz pdb_p01_oiz
10.9.132.61(54140)idle 
77908 postgres  20    0 2266M 1960M sleep  32:13  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(44496)idle 
60355 postgres  20    0 2266M 1864M sleep  29:00  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36638)idle 
92184 postgres  20    0 2187M 1893M sleep  28:20  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(33216)idle 
60356 postgres  20    0 2236M 1852M sleep  27:56  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36642)idle 
60359 postgres  20    0 2283M 1906M sleep  24:03  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36644)idle 
77909 postgres  20    0 2219M 1947M sleep  23:59  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(44498)idle 
77910 postgres  20    0 2191M 1918M sleep  22:20  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(44504)idle 
60357 postgres  20    0 2203M 1935M sleep  20:48  0.52%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36635)idle 
96423 postgres  20    0 2242M 1945M sleep  18:29  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(43288)idle 
60328 postgres  20    0 2222M 1947M sleep  18:23  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36558)idle 
99094 postgres  20    0 2349M 1989M sleep  16:58  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(58596)idle 
60339 postgres  20    0 1774M 1361M sleep  16:10  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36578)idle 
60354 postgres  20    0 2282M 1940M sleep  14:56  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36634)idle 
77703 postgres  20    0 2035M 1714M sleep  12:29  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(45088)idle 
60358 postgres  20    0 1788M  608M sleep  11:43  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.62(36640)idle 
110412 postgres  20    0 2100M  813M sleep  11:26  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(48528)idle 
60317 postgres  20    0 1986M 1634M sleep  11:24  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(40160)idle 
58878 postgres  20    0 2003M 1694M sleep   9:50  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54224)idle 
60315 postgres  20    0 2060M 1805M sleep   9:50  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(40156)idle 
51969 postgres  20    0 2018M  829M sleep   8:54  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54512)idle 
58875 postgres  20    0 1957M 1553M sleep   8:53  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54218)idle 
77704 postgres  20    0 2108M  563M sleep   8:31  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(45090)idle 
58876 postgres  20    0 1925M  831M sleep   8:29  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54220)idle 
58880 postgres  20    0 2236M  807M sleep   8:24  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54228)idle 
58862 postgres  20    0 1684M  559M sleep   7:58  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54158)idle 
77702 postgres  20    0 1027M  712M sleep   7:06  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(45086)idle 
110414 postgres  20    0 1294M  710M sleep   6:54  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(48532)idle 
110413 postgres  20    0 1968M 1641M sleep   6:53  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(48530)idle 
60316 postgres  20    0 2341M  810M sleep   6:52  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(40158)idle 
58877 postgres  20    0 1946M  900M sleep   6:51  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54222)idle 
58851 postgres  20    0 1954M  764M sleep   6:27  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(54138)idle 
10366 postgres  20    0 2028M  730M sleep   6:17  0.00%  0.00% postgres: pcl_p011: pdb_ppp_oiz pdb_p02_oiz
10.9.132.60(49554)idle