On 30/10/2020 18:36, Heikki Linnakangas wrote:
> Whether the leader process finds the EOLs or the worker processes, it's
> pretty clear that it needs to be done ASAP, for a chunk at a time,
> because that cannot be done in parallel. I think some refactoring in
> CopyReadLine() and friends would be in order. It probably would be
> faster, or at least not slower, to find all the EOLs in a block in one
> tight loop, even when parallel copy is not used.
Something like the attached. It passes the regression tests, but it's
quite incomplete. It's missing handing of "\." as end-of-file marker,
and I haven't tested encoding conversions at all, for starters. Quick
testing suggests that this a little bit faster than the current code,
but the difference is small; I had to use a "WHERE false" option to
really see the difference.
The crucial thing here is that there's a new function, ParseLinesText(),
to find all end-of-line characters in a buffer in one go. In this patch,
it's used against 'raw_buf', but with parallel copy, you could point it
at a block in shared memory instead.
- Heikki