Re: Catalog/Metadata consistency during changeset extraction from wal

On Thursday, June 21, 2012 01:41:25 PM Andres Freund wrote:
> Below are two possible implementation strategies for that concept
> 
> Advantages:
> * Decoding is done on the master in an asynchronous fashion
> * low overhead during normal DML execution, not much additional code in
> that  path
> * can be very efficient if architecture/version are the same
> * version/architecture compatibility can be done transparently by falling
> back  to textual versions on mismatch
> 
> Disadvantages:
> * decoding probably has to happen on the master which might not be what
> people  want performancewise

> 3b)
> Ensure that enough information in the catalog remains by fudging the xmin 
> horizon. Then reassemble an appropriate snapshot to read the catalog as
> the  tuple in question has seen it.
> 
> Advantages:
> * should be implementable with low impact to general code
> 
> Disadvantages:
> * requires some complex code for assembling snapshots
> * it might be hard to guarantee that we always have enough information to 
> reassemble a snapshot (subxid overflows ...)
> * impacts vacuum if replication to some site is slow
There are some interesting problems related to locking and snapshots here. Not 
sure if they are resolvable:

We need to restrict SnapshotNow to represent to the view it had back when the 
wal record were currently decoding had. Otherwise we would possibly get wrong 
column types and similar. As were working in the past locking doesn't protect 
us against much here. I have that (mostly and inefficiently).

One interesting problem are table rewrites (truncate, cluster, some ALTER 
TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it 
had back when the wal record was created we get the old relfilenode. Which 
might have been dropped in part of the transaction cleanup...
With most types thats not a problem. Even things like records and arrays  
aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. 
pg_enum) and vacuum full'ing a table which is used in the *_out function of a 
type (like a user level pg_enum implementation).

The only theoretical way I see against that problem would be to postpone all 
relation unlinks untill everything that could possibly read them has finished. 
Doesn't seem to alluring although it would be needed if we ever move more 
things of SnapshotNow.

Input/Ideas/Opinions?

Greetings,

Andres
-- 
Andres Freund        http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services