On Sat, Dec 5, 2015 at 12:10 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 4 December 2015 at 13:47, Ildus Kurbangaliev
> <i.kurbangaliev@postgrespro.ru> wrote:
>
>>
>> Attached patch adds support of partial decompression for datums.
>> It will be useful in many cases when extracting part of data is
>> enough for big varlena structures.
>>
>> It is especially useful for expanded datums, because it provides
>> storage for partial results.
>
>
> This isn't enough for anyone else to follow your thoughts and agree enough
> to commit.
>
> Please explain the whole idea, starting from what problem you are trying to
> solve and how well this does it, why you did it this way and the other ways
> you tried/decided not to pursue. Thanks.
Yeah, I would imagine that what Ildus is trying to achieve is
something close to LZ4_decompress_safe_partial, by being able to stop
compression after getting a certain amount of data decompressed, and
continue working once again after.
And actually I think I get the idea. With his test case, what we get
first is a size, and then we reuse this size to extract only what we
need to fetch only a number of items from the tsvector. But that's
actually linked to the length of the compressed chunk, and at the end
we would still need to decompress the whole string perhaps, but it is
not possible to be sure using the information provided.
Ildus, using your patch for tsvector, are you aiming at being able to
complete an operation by only using a portion of the compressed data?
Or are you planning to use that to improve the speed of detection of
corrupted data in the chunk? If that's the latter, we would need to
still decompress the whole string anyway, so having a routine able to
decompress only until a given position is not necessary, and based on
the example given upthread it is not possible to know what you are
trying to achieve. Hence could you share your thoughts regarding your
stuff with tsvector?
Changing pglz_decompress shape is still a bad idea anyway, I guess we
had better have something new like pglz_decompress_partial instead.
--
Michael