The time has come.
FPGA optimization is in the palm of our hands (literally a 2 TB 40 GB/s IO PostgreSQL server fits into less than a shoe box), and on Amazon AWS F1 instances.
Some demos are beginning to exist: https://github.com/Xilinx/data-analytics.
But a lot more could be done. How about linear sort performance at O(N)? https://hackaday.com/2016/01/20/a-linear-time-sorting-algorithm-for-fpgas/. And how about https://people.csail.mit.edu/wjun/papers/fccm2017.pdf, the following four sorting accelerators are used:
- Tuple Sorter : Sorts an N-tuple using a sorting network.
- Page Sorter : Sorts an 8KB (a flash page) chunk of sorted N-tuples in on-chip memory.
- Super-Page Sorter : Sorts 16 8K-32MB sorted chunks in DRAM.
- Storage-to-Storage Sorter: Sorts 16 512MB or larger sorted chunks in flash.
Order of magnitude speed improvements? Better than Hadoop clusters on a single chip? 40 GB/s I/O throughput massive full table scan, blazing fast sort-merge joins? Here it is. Anybody working more on that? Should be an ideal project for a student or a group of students.
Is there a PostgreSQL foundation I could donate to, 501(c)(3) tax exempt? I can donate and possibly find some people at Purdue University who might take this on. Interest?
regards,
-Gunther