Dear Hackers,
We at Citus Data have been developing a columnar store extension for PostgreSQL. Today we are excited to open source it under the Apache v2.0 license.
This columnar store extension uses the Optimized Row Columnar (ORC) format for its data layout, which improves upon the RCFile format developed at Facebook, and brings the following benefits:
* Compression: Reduces in-memory and on-disk data size by 2-4x. Can be extended to support different codecs. We used the functions in pg_lzcompress.h for compression and decompression.
* Column projections: Only reads column data relevant to the query. Improves performance for I/O bound queries.
* Skip indexes: Stores min/max statistics for row groups, and uses them to skip over unrelated rows.
We used the PostgreSQL FDW APIs to make this work. The extension doesn't implement the writable FDW API, but it uses the process utility hook to enable COPY command for the columnar tables.
This extension uses PostgreSQL's internal data type representation to store data in the table, so this columnar store should support all data types that PostgreSQL supports.
We tried the extension on TPC-H benchmark with 4GB scale factor on a m1.xlarge Amazon EC2 instance, and the query performance improved by 2x-3x compared to regular PostgreSQL table. Note that we flushed the page cache before each test to see the impact on disk I/O.
When data is cached in memory, the performance of cstore_fdw tables were close to the performance of regular PostgreSQL tables.
For more information, please visit:
Feedback from you is really appreciated.
Thanks,
-- Hadi