Re: Proposal for CSN based snapshots

On 2014-05-12 16:56:51 +0300, Heikki Linnakangas wrote:
> On 01/24/2014 02:10 PM, Rajeev rastogi wrote:
> >We are also planning to implement CSN based snapshot.
> >So I am curious to know whether any further development is happening on this.
> 
> I started looking into this, and plan to work on this for 9.5. It's a big
> project, so any help is welcome. The design I have in mind is to use the LSN
> of the commit record as the CSN (as Greg Stark suggested).

Cool.

I haven't fully thought it through but I think it should make some of
the decoding code simpler. And it should greatly simplify the hot
standby code.

Some of the stuff in here will be influence whether your freezing
replacement patch gets in. Do you plan to further pursue that one?

> The core of the design is to store the LSN of the commit record in pg_clog.
> Currently, we only store 2 bits per transaction there, indicating if the
> transaction committed or not, but the patch will expand it to 64 bits, to
> store the LSN. To check the visibility of an XID in a snapshot, the XID's
> commit LSN is looked up in pg_clog, and compared with the snapshot's LSN.

We'll continue to need some of the old states? You plan to use values
that can never be valid lsns for them? I.e. 0/0 IN_PROGRESS, 0/1 ABORTED
etc?
How do you plan to deal with subtransactions?

> Currently, before consulting the clog for an XID's status, it is necessary
> to first check if the transaction is still in progress by scanning the proc
> array. To get rid of that requirement, just before writing the commit record
> in the WAL, the backend will mark the clog slot with a magic value that says
> "I'm just about to commit". After writing the commit record, it is replaced
> with the record's actual LSN. If a backend sees the magic value in the clog,
> it will wait for the transaction to finish the insertion, and then check
> again to get the real LSN. I'm thinking of just using XactLockTableWait()
> for that. This mechanism makes the insertion of a commit WAL record and
> updating the clog appear atomic to the rest of the system.

So it's quite possible that clog will become more of a contention point
due to the doubled amount of writes.

> In theory, we could use a snapshot LSN as the cutoff-point for
> HeapTupleSatisfiesVisibility(). Maybe it's just because this is new, but
> that makes me feel uneasy.

It'd possibly also end up being less efficient because you'd visit the
clog for potentially quite some transactions to get the LSN.

Greetings,

Andres Freund