On Mon, Aug 20, 2018 at 11:09 AM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> 2. Teach your GetForeignPath function to do something like this:
> [blah blah blah]
I was pinged off-list by someone who is working on a parallel-aware
FDW, who asked if I still had the test code I mentioned up-thread.
While digging that out, I couldn't resist hacking it a bit more until
it gave the right answers, only sooner:
$ seq -f '%20.0f' 1 10000000 > numbers.csv
create extension file_fdw;
create server files foreign data wrapper file_fdw;
create foreign table numbers (n int) server files
options (filename '/path/to/numbers.csv', format 'csv');
explain select count(*) from numbers;
select count(*) from numbers;
Non-parallel: 2.6s
1 worker: 1.4s
2 workers: 0.9s
3 workers: 0.7s
Finally, I can do parallel hash joins between CSV files!
select count(*) from numbers n1 join numbers n2 using (n);
Non-parallel: 11.4s
1 worker: 6.6s
2 workers: 4.8s
3 workers: 4.1s
There are probably some weird fence-post or accounting bugs hiding in
this patch -- it has to count bytes carefully, and deal with some edge
cases around lines that span chunks. It's only a rough draft, but
might eventually serve as a useful example of a parallel-aware FDW.
--
Thomas Munro
https://enterprisedb.com