A question to all pg hackers
Is anybody working on adding pipelining to set returning functions.
How much effort would it take ?
Where should I start digging ?
BACKGROUND:
AFAICS , currently set returning functions materialise their results
before returning, as seen by this simple test:
hannu=# select * from generate_series(1,10) limit 2;generate_series
----------------- 1 2
(2 rows)
Time: 1.183 ms
hannu=# select * from generate_series(1,10000000) limit 2;generate_series
----------------- 1 2
(2 rows)
Time: 3795.032 ms
being able to pipeline (generate results as needed) would enable several
interesting techniques, especially if combined with pl/proxy or any
other functions which stream external data.
Applications and design patterns like http://telegraph.cs.berkeley.edu/
or http://labs.google.com/papers/mapreduce.html would suddenly become
very easy to implement.
-----------------
Hannu