Обсуждение: OperationalError: FATAL: could not open file : Too many open files in system

Hello,

I'm managing a pgsql->django->mod_python->apache machine that serves no media, but handles postgres, memcached and apache all by itself.

Recently I've begun drawing "too many open files in the system" errors, as seen in the subject line.

Browsing the postgres documentation, I suspect that I need to lower my `max_files_per_process` from the default 100. But the support team at my host is pointing the finger at memcached.

Before I do anything, I thought I'd consult the experts here on the list and see whether you can had any opinions. Here are some of the basic config settings that, from cruising around the web and talking to listservs, I've come to suspect are crucial to this sort of problem. Really, any advice is appreciated, because I've been getting competiting advice from different sources and I'm not sure exactly where to go with this.

And, of course, thanks listers and developers for such a great product and community.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 137215
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

postgres.conf (max_files_per_process is commented out)
max_connections = 100
shared_buffers = 1000

$ httpd -l

Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

httpd.conf...

Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

$ cat /proc/sys/fs/file-nr
73320   0       262144

$ lsof | awk '{print $1}' | sort | uniq -c
      1 COMMAND
     10 awk
     49 bash
     17 crond
   5154 httpd
      9 init
     25 lsof
     39 memcached
    807 postmaste
     62 saslauthd
     69 sendmail
     28 sh
      7 sort
    183 sshd
     13 syslogd
      8 uniq
     19 xinetd

On Apr 23, 2009, at 1:39 PM, Ben Welsh wrote:

> Hello,
>
> I'm managing a pgsql->django->mod_python->apache machine that serves
> no media, but handles postgres, memcached and apache all by itself.
>
> Recently I've begun drawing "too many open files in the system"
> errors, as seen in the subject line.
>
> Browsing the postgres documentation, I suspect that I need to lower
> my `max_files_per_process` from the default 100. But the support
> team at my host is pointing the finger at memcached.
>
> Before I do anything, I thought I'd consult the experts here on the
> list and see whether you can had any opinions. Here are some of the
> basic config settings that, from cruising around the web and talking
> to listservs, I've come to suspect are crucial to this sort of
> problem. Really, any advice is appreciated, because I've been
> getting competiting advice from different sources and I'm not sure
> exactly where to go with this.
>
> And, of course, thanks listers and developers for such a great
> product and community.
>

<snip>

> $ lsof | awk '{print $1}' | sort | uniq -c
>       1 COMMAND
>      10 awk
>      49 bash
>      17 crond
>    5154 httpd
>       9 init
>      25 lsof
>      39 memcached
>     807 postmaste
>      62 saslauthd
>      69 sendmail
>      28 sh
>       7 sort
>     183 sshd
>      13 syslogd
>       8 uniq
>      19 xinetd

I'd start by investigating the 5K+ file descriptors held by Apache.

Erik Jones, Database Administrator
Engine Yard
Support, Scalability, Reliability
866.518.9273 x 260
Location: US/Pacific
IRC: mage2k