"Matt Magoffin" <postgresql.org@msqr.us> writes:
> I just noticed something: the "open files" limit lists 1024, which is the
> default for this system. A quick count of open data files currently in use
> by Postgres returns almost 7000, though.
In principle, any problem with that ought to result in explicit error
messages like "too many open files". In practice, what can happen is
that Postgres can run the kernel out of file table slots (which are a
finite system-wide resource) and then everything else on the box starts
to fall over. Postgres itself will often survive just fine, since it
has a preprogrammed response to ENFILE to close some other file it
doesn't need right now and try again. Most other userland code will
just croak if open() starts failing.
Which is a longwinded way of saying "that doesn't seem to match the
symptoms, but ..." If you're not dead certain that your kernel is
configured to allow *well* north of 7000 open files, you might consider
cutting max_files_per_process in half at your next restart. I am
wondering if there could be some kind of memory-pressure effect inside
the kernel proper that causes lots of open files to translate into
unexpected denials of requests for more process memory. It's kind of
grasping at straws, admittedly. But it seems like we've eliminated the
possibility of a material memory leak inside the Postgres backend
process (even if you don't trust the memory context dump on its own,
the reported process VSIZE backs it up), and the other printout you
showed said that there was hardly any usage of swap space, which seems
to defeat the theory that there's system-wide memory pressure in the
normal sense.
OTOH ... wait a minute. Have you maybe got the system configured to
start denying memory requests before it gets into significant swapping?
We typically suggest setting vm.overcommit_memory=2 on Linux, but
I'm not sure whether that results in the kernel trying to avoid using
swap at all. Maybe this really is an issue with system-wide total
memory use rather than any one process.
regards, tom lane