Обсуждение: Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

Hi,

On 2025-02-11 19:52:34 +0100, Jelte Fennema-Nio wrote:
> So this starts bumping postmaster and pgbench its soft open file limit
> to the hard open file limit.

Not sure that's quite the right thing to do for postmaster. What I'd start
with is to increase the soft limit to
  "already used files" + max_files_per_process.

That way we still limit "resource" usage, but react better to already used
FDs. If io_uring, listen_addresses, whatnot use FDs
max_files_per_process would be added ontop.

Then we can separately discuss increasing max_files_per_process more
aggressively.


I don't see a downside to just increasing the soft limit for pgbench. It
avoids the stupid cycle of getting "need at least %d open files, but system
limit is %ld", increase ulimit, retry, without any non-theoretical downsides.


> Doing so is especially useful for the AIO work that Andres is doing, because
> io_uring consumes a lot of file descriptors.

Yep.


One more reason this is a good idea is that we'll also need this for
threading, since there all client connections obviously will eat into the
"normal file descriptor" budget.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I don't see a downside to just increasing the soft limit for
> pgbench.

Agreed, that end of the patch seems relatively harmless.

            regards, tom lane

On Tue Feb 11, 2025 at 8:42 PM CET, Andres Freund wrote:
> Not sure that's quite the right thing to do for postmaster. What I'd start
> with is to increase the soft limit to
>   "already used files" + max_files_per_process.

I needed to rebase this patch, and that made me finally take the time to
do the restoration of the file limit for subprocesses properly: In
previous versions of this patch it restored the limit before the call to
system() and it didn't restore it at all for popen. This latest version
the patch adds custom pg_system() and pg_popen() functions that restore
the limits in the child process right after the fork, but before the
exec.

There are two reasons to do this:
1. Any executables that still use select(2) will get clear "out of file
   descriptors" errors instead of failing in mysterious ways.
2. Future looking when we'll have multi-threading (which this change is
   needed for) it would be problematic to restore the original limit
   temporarily in the postgres process tree. Some other thread might
   want to open a file while the limit is too low. By only calling
   setrlimit with the lower value in the child process there's not a
   single moment where the original Postgres process has a too low limit.