Re: [PATCH] Infinite loop while acquiring new TOAST Oid

On Mon, Nov 28, 2022 at 05:24:29PM -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
>> And as you said earlier, the increased overhead inside the toast table / index
>> is not relevant compared to the size of toasted datums.
>
> Perhaps not.

(Replying to an old thread here, due to some work involving more
compression methods in this area.)

> I agree that we can't simply widen varatt_external to use 8 bytes for
> the toast ID in all cases.  Also, I now get the point about avoiding
> use of globally assigned OIDs here: if the counter starts from zero
> for each table, then a variable-width varatt_external could actually
> be smaller than currently for many cases.  However, that bit is somewhat
> orthogonal, and it's certainly not required for fixing the basic problem.
>
> So it seems like the plan of attack ought to be:
>
> 1. Invent a new form or forms of varatt_external to allow different
> widths of the toast ID.  Use the narrowest width possible for any
> given ID value.
>
> 2. Allow TOAST tables/indexes to store either 4-byte or 8-byte IDs.
> (Conversion could be done as a side effect of table-rewrite
> operations, perhaps.)

> 3. Localize ID selection so that tables can have small toast IDs
> even when other tables have many IDs.  (Optional, could do later.)

Hmm, I was reading this thread, and I am wondering if tackling the
problem in the opposite order is not actually the path of least pain.
Doing 3 first where we would switch from a global source assigning the
toast IDs to a local source for each toast relation can lift quite a
bit of the pain we have, especially if we rely on a sequence attached
to the toast table to do all the value assignment work (WAL logging,
already 8 bytes, etc.).  A new vartag_external would still be required
to store a 4-byte toast ID, and it could always be extended to 8 byte
if necessary.  That sounds like having more benefits.
--
Michael