Re: postmaster uses more CPU in 18 beta1 with io_method=io_uring

On Tue, Jul 8, 2025 at 5:22 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-06-30 12:27:10 -0400, Andres Freund wrote:
> > On 2025-06-05 14:32:10 -0400, Andres Freund wrote:
> > > On 2025-06-05 12:47:52 -0400, Tom Lane wrote:
> > > > Andres Freund <andres@anarazel.de> writes:
> > > > > I think this is a big enough pitfall that it's, obviously assuming the patch
> > > > > has a sensible complexity, worth fixing this in 18. RMT, anyone, what do you
> > > > > think?
> > > >
> > > > Let's see the patch ... but yeah, I'd rather not ship 18 like this.
> > >
> > > I've attached a first draft.
> > >
> > > I can't make heads or tails of the ordering in configure.ac, so the function
> > > test is probably in the wrong place.
> >
> > Any comments on that patch?  I'd hoped for some review comments... Unless I'll
> > hear otherwise, I'll just do a bit more polish and push..
>
> After addressing most of Greg's and Jim's feedback, I pushed this. I chose not
> to increase the log level as Jim suggested, but if we end up deciding that
> that's the way to go, we can easily change that...
>

Hi Andres,

I'm with Jim as I've just hit it but not on exit() but for fork(), so:

1. Could we s/DEBUG1/INFO/ that debug message level? (for those two:
"cannot use combined memory mapping for io_uring" , and maybe add
"potential slow new connections" there too along the way?)
2. Maybe we could add some wording to the docs about io_method that it
might cause such trouble ?

Just wasted an hour on wondering why $stuff is slow, given:
    max_connections = '20000' # yes, yay..
    io_method = 'io_uring'

I was getting like slow fork()/clone() performance when there's were
lots of io_uring fds/instances in the main postmaster:
    $ /usr/pgsql19/bin/pgbench -f select1.sql -c 1000 -j 1 -t 1 -P 1
    [..]
    progress: 39.7 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
    progress: 40.6 s, 1039.9 tps, lat 407.696 ms stddev 291.856, 0 failed
    [..]
    initial connection time = 39632.164 ms
    tps = 1015.608893 (without initial connection time)

So yes, ~40s to just connect to the database and I was using some old
branch from back before Jun (it was not having f54af9f2679d5987b46),
so simulating <= 6.5 as You say more or less. I was limited to 20-30
forks()/1sec according to bpftrace. It goes away with default
io_method (~800 forks()/1sec). With max_connections = 2k, I got 5s
initial connection times. It looked like caused by io_uring, as with
io_uring fork() was slow somewhere in vma_interval_tree_insert_after
<- copy_process <- kernel_clone <- __do_sys_clone <- do_syscall_64
(?). I've tested it on 6.14.17 too, but also on LTS 6.1.x too (well
the difference is that it takes 65s instead of 40s...). Then searched
and hit this thread, but 6.1 is the LTS kernel, so plenty of people
are going to hit those regressions with io_uring io_method, won't
they?

I can try to prepare a patch, please just let me know.

-J.