On Tue, 18 Feb 2020 at 04:40, Thomas Munro <thomas.munro@gmail.com> wrote:
> +1. That sort of two-queue scheme is exactly how I sketched out a
> multi-consumer queue for a hypothetical Parallel Scatter node. It
> probably gets a bit trickier when the payload has to be broken up into
> fragments to wrap around the "data" buffer N times.
At least for copy it should be easy enough - it already has to handle reading
data block by block. If worker updates its position while doing so the reader
can wrap around the data buffer.
There will be no parallelism while one worker is buffering up a line larger
than the data buffer, but that doesn't seem like a major issue. Once the line is
buffered and begins inserting next worker can start buffering the next tuple.
Regards,
Ants Aasma