Re: PGLZ Compression Optimization

Re: PGLZ Compression Optimization

> On 16 Jul 2026, at 19:06, Tomas Vondra  wrote:
> 
> But making the current scheme
> faster (particularly the decompression part) would be a huge benefit for
> existing systems that can't simply recompress everything.

FWIW this spring we took a stab at refactoring old patch [0] about optimizing pglz
decompression ideas into several small patches [1] [2]. We stalled when found some
regressions and work lost momentum.

If anyone is interested - we (me, Nik, Kirk) would be happy to collab. Some enthusiasm
could revive that work.


Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/flat/FEF3DC5E-4BC4-44E1-8DEB-DADC67046FE3%40yandex-team.ru
[1] https://github.com/NikolayS/pglz-compression/pull/20
[2] https://github.com/NikolayS/pglz-compression/pull/20/changes/dcb12183519c2eda76dfadaac005e12de6ad4e01

Re: PGLZ Compression Optimization

> On 17 Jul 2026, at 10:17, wenhui qiu  wrote:

> Thank you for providing this information; I’ll take a close look at it.LZ4 is
> just too good—no matter how hard I try to improve PGLZ, it can’t even come
> close to it.

That's probably true, but given that there are lots of clusters using pglz in
production, if we could shave x% of time it takes to decode that could still
be worthwhile.

--
Daniel Gustafsson

Re: PGLZ Compression Optimization

On 7/16/26 14:16, Daniel Gustafsson wrote:
>> On 13 Jul 2026, at 20:17, Tomas Vondra  wrote:
>> On 7/13/26 13:15, wenhui qiu wrote:
> 
>>> The patch also optimizes the legacy path with a rolling hash, indexed
>>> history rings, bounded candidate searches, word/SIMD match extension,
>>> and an AVX2 implementation selected by PostgreSQL's runtime CPU check.
>>> Other platforms use PostgreSQL's existing SIMD abstraction.
>>>
>>
>> This, on the other hand would be a *huge* improvement for all existing
>> systems with a lot of data compressed with pglz (and there's fair amount
>> of those). If we could just make them faster by optimizing the code,
>> that seems very desirable.
> 
> +1, if optimizing the code can unlock performance improvements while being
> backwards compatible it would be a very big win.
> 

100% agreed. Introducing new compression schemes into PGLZ is one thing
(and I think it'd be wasted effort). But making the current scheme
faster (particularly the decompression part) would be a huge benefit for
existing systems that can't simply recompress everything.

I wonder if we could introduce a mechanism to automatically recompress
stuff in the background (with a background worker, similar to what
online checksums do). That'd allow systems to migrate to a new
compression algorithm. But that's a completely separate feature.

regards

-- 
Tomas Vondra

Re: PGLZ Compression Optimization

> On 13 Jul 2026, at 20:17, Tomas Vondra  wrote:
> On 7/13/26 13:15, wenhui qiu wrote:

>> The patch also optimizes the legacy path with a rolling hash, indexed
>> history rings, bounded candidate searches, word/SIMD match extension,
>> and an AVX2 implementation selected by PostgreSQL's runtime CPU check.
>> Other platforms use PostgreSQL's existing SIMD abstraction.
>> 
> 
> This, on the other hand would be a *huge* improvement for all existing
> systems with a lot of data compressed with pglz (and there's fair amount
> of those). If we could just make them faster by optimizing the code,
> that seems very desirable.

+1, if optimizing the code can unlock performance improvements while being
backwards compatible it would be a very big win.

--
Daniel Gustafsson

Re: PGLZ Compression Optimization

Hello!

On 7/13/26 13:15, wenhui qiu wrote:
> HI  hackers
>
> LZ4 and Zstandard are both excellent and well-maintained open-source
> projects.  However, they remain optional dependencies maintained outside
> the PostgreSQL project, while PGLZ is an implementation that PostgreSQL
> ships and controls itself.
>
> Given the known performance gap between PGLZ and modern compression
> algorithms, I wanted to explore a simple question: rather than relying
> only on external libraries for better compression performance, can we
> also improve PostgreSQL's own built-in compression algorithm?
>
> This patch grew out of that idea.  It is not intended to replace LZ4 or
> Zstandard, but to make PostgreSQL's self-contained compression option
> substantially more competitive.
>
>
> The attached RFC patch explores a self-contained modernization of PGLZ.
>
> For inputs of at least 32KB, the encoder first tries a new byte-aligned
> stream format.  It uses a single-candidate LZ77 parser, a 65535-byte
> history window, literal runs, fixed-width offsets, and ULEB128 length
> extensions.  This format is designed primarily for fast decoding.
>
> If that representation cannot satisfy the compression strategy, a second
> format can encode literals with canonical Huffman codes.  Smaller inputs
> and unsuccessful modern-format attempts continue to use the legacy PGLZ
> representation.
>

So it effectively implements a new compression algorithm/format,
incompatible with PGLZ? Considering there are great external compression
libraries, why should we try to compete with them? I'm sure inventing
new compression algorithm is fun, but ISTM we're unlikely to beat the
external libraries.

I suppose most packages are built with these external libraries, right?
So people who care about compression would already benefit from that.
We've even switched TOAST to lz4 when possible in PG 19.

> The patch also optimizes the legacy path with a rolling hash, indexed
> history rings, bounded candidate searches, word/SIMD match extension,
> and an AVX2 implementation selected by PostgreSQL's runtime CPU check.
> Other platforms use PostgreSQL's existing SIMD abstraction.
>

This, on the other hand would be a *huge* improvement for all existing
systems with a lot of data compressed with pglz (and there's fair amount
of those). If we could just make them faster by optimizing the code,
that seems very desirable.

> The new streams begin with a sequence that is invalid in the legacy
> format, allowing the decoder to distinguish them without changing the
> external TOAST compression identifier.  The new decoder continues to
> read existing legacy PGLZ values.  Compatibility is one-way: an
> unpatched server cannot decode values written in either new format.
>
> The implementation is self-contained.  It does not link to or incorporate
> source code from LZ4 or Zstandard.
>
> I benchmarked the 51,975-byte example-large.json input, repeated 4,100
> times per run.  The following are medians of five runs on ARM64 macOS,
> compiled with Apple clang -O3, based on master at 1ae87df63ff:
>
>                          master       patched          LZ4
>   compression MB/s        325.9        2137.3        2811.2
>   decompression MB/s     2831.1        7582.5       11777.8
>   compressed bytes        14370         12445         12802
>
> On this input, the patch made PGLZ compression about 6.6 times faster
> and decompression about 2.7 times faster than master.  The resulting
> stream was 13.4% smaller.  It remains slower than LZ4, especially during
> decompression.
>

Those are nice improvements, but once again - why wouldn't I just use
lz4 for new system? The new algorithm still doesn't beat it, right?

> Testing performed:
>
> * clean out-of-tree build with --with-lz4 and --with-zstd
> * make check: all 245 tests passed
> * 24,820 full and partial round-trip cases under ASan and UBSan
> * tests for partial decompression and malformed version-3 streams
> * git diff --check
>
> This is intentionally an RFC because it changes the PGLZ on-disk stream
> and is currently a large patch.  I would especially appreciate feedback
> on:
>
> * whether the one-way compatibility model is acceptable
> * whether having both the byte-aligned and Huffman fallback formats is
>   worth the implementation complexity
> * whether the 32KB selection threshold is reasonable

I think one-way compatibility would be fine, it'd require a major
version upgrade anyway. But I'm not convinced we want to extend the pglz
like this.

> * whether the legacy hot-path changes should be submitted separately
>

Yes. I think the cost/benefit is much clearer there, and I expect this
part to have much higher chance of succeeding.


> For transparency, I used an AI coding assistant while prototyping parts
> of this patch.  I reviewed and tested the resulting implementation and
> took responsibility for maintaining and revising it.
>
> Thanks


regards

-- 
Tomas Vondra

Re: PGLZ Compression Optimization

PGLZ Compression Optimization

   LZ4 and Zstandard are both excellent and well-maintained open-source
projects.  However, they remain optional dependencies maintained outside
the PostgreSQL project, while PGLZ is an implementation that PostgreSQL
ships and controls itself.

Given the known performance gap between PGLZ and modern compression
algorithms, I wanted to explore a simple question: rather than relying
only on external libraries for better compression performance, can we
also improve PostgreSQL's own built-in compression algorithm?

This patch grew out of that idea.  It is not intended to replace LZ4 or
Zstandard, but to make PostgreSQL's self-contained compression option
substantially more competitive.
The attached RFC patch explores a self-contained modernization of PGLZ.

For inputs of at least 32KB, the encoder first tries a new byte-aligned
stream format.  It uses a single-candidate LZ77 parser, a 65535-byte
history window, literal runs, fixed-width offsets, and ULEB128 length
extensions.  This format is designed primarily for fast decoding.

If that representation cannot satisfy the compression strategy, a second
format can encode literals with canonical Huffman codes.  Smaller inputs
and unsuccessful modern-format attempts continue to use the legacy PGLZ
representation.

The patch also optimizes the legacy path with a rolling hash, indexed
history rings, bounded candidate searches, word/SIMD match extension,
and an AVX2 implementation selected by PostgreSQL's runtime CPU check.
Other platforms use PostgreSQL's existing SIMD abstraction.

The new streams begin with a sequence that is invalid in the legacy
format, allowing the decoder to distinguish them without changing the
external TOAST compression identifier.  The new decoder continues to
read existing legacy PGLZ values.  Compatibility is one-way: an
unpatched server cannot decode values written in either new format.

The implementation is self-contained.  It does not link to or incorporate
source code from LZ4 or Zstandard.

I benchmarked the 51,975-byte example-large.json input, repeated 4,100
times per run.  The following are medians of five runs on ARM64 macOS,
compiled with Apple clang -O3, based on master at 1ae87df63ff:
                         master       patched          LZ4  compression MB/s        325.9        2137.3        2811.2  decompression MB/s     2831.1        7582.5       11777.8  compressed bytes        14370         12445         12802

On this input, the patch made PGLZ compression about 6.6 times faster
and decompression about 2.7 times faster than master.  The resulting
stream was 13.4% smaller.  It remains slower than LZ4, especially during
decompression.

Testing performed:

* clean out-of-tree build with --with-lz4 and --with-zstd
* make check: all 245 tests passed
* 24,820 full and partial round-trip cases under ASan and UBSan
* tests for partial decompression and malformed version-3 streams
* git diff --check

This is intentionally an RFC because it changes the PGLZ on-disk stream
and is currently a large patch.  I would especially appreciate feedback
on:

* whether the one-way compatibility model is acceptable
* whether having both the byte-aligned and Huffman fallback formats is  worth the implementation complexity
* whether the 32KB selection threshold is reasonable
* whether the legacy hot-path changes should be submitted separately

For transparency, I used an AI coding assistant while prototyping parts
of this patch.  I reviewed and tested the resulting implementation and
took responsibility for maintaining and revising it.

Re: PGLZ Compression Optimization

Re: PGLZ Compression Optimization

Hi,

On 7/16/26 11:08, wenhui qiu wrote:
> Hi Tomas
> 
>     Thank you for your reply , Why did I come up with this idea? When I
>     suggested changing `wal_compression on` to  ZSTD or LZ4,  Wasn’t
>     accepted. So I wondered if it could be implemented internally. In
>     any case, it can’t compete with LZ4, so I don’t think it’s worth
>     investing effort into improving PG-LZ. Setting LZ4 or ZSTD as the
>     default is the best option.
> 

Maybe I'm missing something, but I did check the wal_compression thread
[1], and I don't think it was rejected at all. I think people agree with
the proposa to change what 'on' means, depending on whether lz4/zstd is
supported by the build. And I agree with that plan, so let me comment on
that thread.

[1]
https://www.postgresql.org/message-id/flat/CAGjGUAL1b%3DMwd1SCvLbo%2BfivEr9KDpFcu4jmqKCZXwT%3D6CiiGQ%40mail.gmail.com


-- 
Tomas Vondra