Re: Parallel copy

On Tue, Feb 18, 2020 at 5:59 PM Ants Aasma  wrote:
>
> On Tue, 18 Feb 2020 at 12:20, Amit Kapila  wrote:
> > This is something similar to what I had also in mind for this idea.  I
> > had thought of handing over complete chunk (64K or whatever we
> > decide).  The one thing that slightly bothers me is that we will add
> > some additional overhead of copying to and from shared memory which
> > was earlier from local process memory.  And, the tokenization (finding
> > line boundaries) would be serial.  I think that tokenization should be
> > a small part of the overall work we do during the copy operation, but
> > will do some measurements to ascertain the same.
>
> I don't think any extra copying is needed.
>

I am talking about access to shared memory instead of the process
local memory.  I understand that an extra copy won't be required.

> The reader can directly
> fread()/pq_copymsgbytes() into shared memory, and the workers can run
> CopyReadLineText() inner loop directly off of the buffer in shared memory.
>

I am slightly confused here.  AFAIU, the for(;;) loop in
CopyReadLineText is about finding the line endings which we thought
that the reader process will do.



-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com