Re: AIO v2.0

On Mon, 30 Sept 2024 at 16:49, Andres Freund <andres@anarazel.de> wrote:
> On 2024-09-17 11:08:19 -0700, Noah Misch wrote:
> > > - I am worried about the need for bounce buffers for writes of checksummed
> > >   buffers. That quickly ends up being a significant chunk of memory,
> > >   particularly when using a small shared_buffers with a higher than default
> > >   number of connection. I'm currently hacking up a prototype that'd prevent us
> > >   from setting hint bits with just a share lock. I'm planning to start a
> > >   separate thread about that.
> >
> > AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
> > Doing better is nice, but I don't consider this a blocker.  I recommend
> > dealing with the worry by reducing the limit initially (128 blocks?).  Can
> > always raise it later.
>
> On storage that has nontrivial latency, like just about all cloud storage,
> even 256 will be too low. Particularly for checkpointer.
>
> Assuming 1ms latency - which isn't the high end of cloud storage latency - 256
> blocks in flight limits you to <= 256MByte/s, even on storage that can have a
> lot more throughput. With 3ms, which isn't uncommon, it's 85MB/s.

FYI, I think you're off by a factor 8, i.e. that would be 2GB/sec and
666MB/sec respectively, given a normal page size of 8kB and exactly
1ms/3ms full round trip latency:

1 page/1 ms * 8kB/page * 256 concurrency = 256 pages/ms * 8kB/page =
2MiB/ms ~= 2GiB/sec.
for 3ms divide by 3 -> ~666MiB/sec.

Kind regards,

Matthias van de Meent
Neon (https://neon.tech)