Re: Zedstore - compressed in-core columnar storage

On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote:
>   On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra
>   <tomas.vondra@2ndquadrant.com> wrote:
>
>     ...
>
>     I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it
>     means "this is a container". And then have another flag to track whether
>     the container is compressed or not. But as I suggested elsewhere in this
>     thread, I think it might be better to store some ID of the compression
>     algorithm used instead of a simple flag.
>
>     FWIW when I had to deal with incremental compression (adding data into
>     already compressed buffers), which is what seems to be happening here, I
>     found it very useful/efficient to allow partially compressed buffers and
>     only trigger recompressin when absolutely needed.
>
>     Applied to this case, the container would first store compressed chunk,
>     followed by raw (uncompressed) data. Say, like this:
>
>     ZSContainerData {
>
>         // header etc.
>
>         int nbytes;         /* total bytes in data */
>         int ncompressed;    /* ncompressed <= nbytes, fully compressed when
>                              * (ncompressed == nbytes) */
>
>         char data[FLEXIBLE_ARRAY_MEMBER];
>     }
>
>     When adding a value to the buffer, it'd be simply appended to the data
>     array. When the container would grow too much (can't fit on the page or
>     something), recompression is triggered.
>
>   I think what you suggested here is exactly how its handled currently, just
>   the mechanics are little different. Plain items are added to page as
>   insertions are performed. Then when page becomes full, compression is
>   triggerred container item is created for them to store the compressed
>   data. Then new insertions are stored as plain items, once again when page
>   becomes full, they are compressed and container item created for it. So,
>   never, compressed data is attempted to be compressed again. So, on page
>   plain items are acting as data section you mentioned above. A page can
>   have mix of n plain and n container items.

Maybe. I'm not going to pretend I fully understand the internals. Does
that mean the container contains ZSUncompressedBtreeItem as elements? Or
just the plain Datum values?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services