Hi hackers!
Here is the patch set rebased onto current master (15 rel beta 2 with commit from 29.06).
Just to remind:
In Pluggable TOAST we suggest a way to make TOAST pluggable as Storage (in a way like Pluggable Access Methods) - we extracted
TOAST mechanics from Heap AM, and made it an independent pluggable and extensible part with our freshly developed TOAST API.
With this patch set you will be able to develop and plug in your own TOAST mechanics for table columns. Knowing internals and/or workflow and workload
of data being TOASTed makes Custom Toasters much more efficient in performance and storage.
We keep backwards compatibility and default TOAST mechanics works as it worked previously, working silently with any Toastable datatype
(and TOASTed values and tables from previous versions, no changes in this) and set as default Toaster is not stated otherwise, but through our TOAST API.
TOAST API does not have any noticeable overhead in comparison to the original (master). Proofs in our research materials (measured).
We've already presented out work at HighLoad, PgCon and PgConf conferences, you can find materials here
We have ready to plug in extension Toasters
- bytea appendable toaster for bytea datatype (impressive speedup with bytea append operation)
- JSONB toaster for JSONB (very cool performance improvements when dealing with TOASTed JSONB)
and prototype Toasters (in development) for PostGIS (much faster then default with geometric data), large binary objects
(like pg_largeobject, but much, much larger, and without existing large object limitations), default Toaster implementation without using Indexes.
Patch set consists of 9 incremental patches:
0001_create_table_storage_v4.patch - SQL syntax fix for CREATE TABLE clause, processing SET STORAGE... correctly;
0002_toaster_interface_v7.patch - TOAST API interface and SQL syntax allowing creation of custom Toaster (CREATE TOASTER ...)
and setting Toaster to a table column (CREATE TABLE t (data bytea STORAGE EXTERNAL TOASTER bytea_toaster);)
0003_toaster_default_v6.patch - Default TOAST implemented via TOAST API;
0004_toaster_snapshot_v6.patch - refactoring of Default TOAST and support for versioned Toast rows;
0005_bytea_appendable_toaster_v6.patch - contrib module bytea_appendable_toaster - special Toaster for bytea datatype with customized append operation;
0006_toasterapi_docs_v2.patch - documentation package for Pluggable TOAST;
0007_fix_alignment_of_custom_toast_pointers_v2.patch - fixes custom toast pointer's
alignment required by bytea toaster by Nikita Glukhov;
0008_fix_toast_tuple_externalize_v2.patch - fixes toast_tuple_externalize function
not to call toast if old data is the same as new one.
0009_bytea_contrib_and_varlena_v1.patch - several late fixups for 0005.
This patch set opens the following issues:
1) With TOAST independent of AM it is used by it makes sense to move compression from AM into Toaster and make Compression one of Toaster's options.
Actually, Toasters allow to use any compression methods independently of AM;
2) Implement default Toaster without using Indexes (currently in development)?
3) Allows different, SQL-accessed large objects of almost infinite size IN DATABASE, unlike current large_object functionality and does not limit their quantity;
4) Several already developed Toasters show impressive results for datatypes they were designed for.
We're gladly appreciate your feedback!
--
Nikita Malakhov
Postgres Professional