Re: [HACKERS] Custom compression methods

On Tue, 21 Nov 2017 18:47:49 +0100
Tomas Vondra  wrote:

> 
> I propose to use either
> 
>    CompressionMethodOptions (and CompressionMethodRoutine)
> 
> or
> 
>    CompressionOptions (and CompressionRoutine)

Sounds good, thanks.

> 
> OK. But then I don't understand why tsvector.c does things like
> 
>     VARSIZE(data) - VARHDRSZ_CUSTOM_COMPRESSED - arrsize
>     VARRAWSIZE_4B_C(data) - arrsize
> 
> instead of
> 
>     VARSIZE_ANY_EXHDR(data) - arrsize
>     VARSIZE_ANY(data) - arrsize
> 
> Seems somewhat confusing.
> 

VARRAWSIZE_4B_C returns original size of data, before compression (from
va_rawsize in current postgres, and from va_info in my patch), not size
of the already compressed data, so you can't use VARSIZE_ANY here.

VARSIZE_ANY_EXHDR in current postgres returns VARSIZE-VARHDRSZ, despite
the varlena is compressed or not, so I just kept this behavior for
custom compressed varlenas too. If you look into tuptoaster.c you will
also see lines like 'VARSIZE(attr) - TOAST_COMPRESS_HDRSZ'. So I think
if VARSIZE_ANY_EXHDR will subtract different header sizes then it
should subtract them for usual compressed varlenas too.

> >   
> 
> Hmmm, it still doesn't work for me. See this:
> 
>     test=# create extension pg_lz4 ;
>     CREATE EXTENSION
>     test=# create table t_lz4 (v text compressed lz4);
>     CREATE TABLE
>     test=# create table t_pglz (v text);
>     CREATE TABLE
>     test=# insert into t_lz4 select repeat(md5(1::text),300);
>     INSERT 0 1
>     test=# insert into t_pglz select * from t_lz4;
>     INSERT 0 1
>     test=# drop extension pg_lz4 cascade;
>     NOTICE:  drop cascades to 2 other objects
>     DETAIL:  drop cascades to compression options for lz4
>     drop cascades to table t_lz4 column v
>     DROP EXTENSION
>     test=# \c test
>     You are now connected to database "test" as user "user".
>     test=# insert into t_lz4 select repeat(md5(1::text),300);^C
>     test=# select * from t_pglz ;
>     ERROR:  cache lookup failed for compression options 16419
> 
> That suggests no recompression happened.

I will check that. Is your extension published somewhere?