Hi,
On 2020-04-10 07:40:06 -0400, Robert Haas wrote:
> On Thu, Apr 9, 2020 at 4:00 PM Andres Freund <andres@anarazel.de> wrote:
> > Imo, yes, there should be only one process doing the chunking. For ilp, cache efficiency, but also because the
leaderis the only process with access to the network socket. It should load input data into one large buffer that's
sharedacross processes. There should be a separate ringbuffer with tuple/partial tuple (for huge tuples) offsets.
Workerprocesses should grab large chunks of offsets from the offset ringbuffer. If the ringbuffer is not full, the
workerchunks should be reduced in size.
>
> My concern here is that it's going to be hard to avoid processes going
> idle. If the leader does nothing at all once the ring buffer is full,
> it's wasting time that it could spend processing a chunk. But if it
> picks up a chunk, then it might not get around to refilling the buffer
> before other processes are idle with no work to do.
An idle process doesn't cost much. Processes that use CPU inefficiently
however...
> Still, it might be the case that having the process that is reading
> the data also find the line endings is so fast that it makes no sense
> to split those two tasks. After all, whoever just read the data must
> have it in cache, and that helps a lot.
Yea. And if it's not fast enough to split lines, then we have a problem
regardless of which process does the splitting.
Greetings,
Andres Freund