Re: POC: Cleaning up orphaned files using undo logs

Antonin Houska <ah@cybertec.at> wrote:

> Amit Kapila <amit.kapila16@gmail.com> wrote:

> > I think we also need to maintain oldestXidHavingUndo for CLOG truncation and
> > transaction-wraparound. We can't allow CLOG truncation for the transaction
> > whose undo is not discarded as that could be required by some other
> > transaction.
>
> Good point. Even the discard worker might need to check the transaction status
> when deciding whether undo log of that transaction should be discarded.

In the zheap code [1] I see that DiscardWorkerMain() discards undo log up to
OldestXmin:


    OldestXmin = GetOldestXmin(NULL, PROCARRAY_FLAGS_AUTOVACUUM |
                               PROCARRAY_FLAGS_VACUUM);

    oldestXidHavingUndo = GetXidFromEpochXid(pg_atomic_read_u64(&ProcGlobal->oldestXidWithEpochHavingUndo));

    /*
     * Call the discard routine if there oldestXidHavingUndo is lagging
     * behind OldestXmin.
     */
    if (OldestXmin != InvalidTransactionId &&
        TransactionIdPrecedes(oldestXidHavingUndo, OldestXmin))
    {
        UndoDiscard(OldestXmin, &hibernate);

and that UndoDiscard() eventually advances oldestXidHavingUndo in the shared
memory.

I'm not sure this is correct because, IMO, OldestXmin can advance as soon as
AbortTransaction() has cleared both xid and xmin fields of the transaction's
PGXACT (by calling ProcArrayEndTransactionInternal). However the corresponding
undo log may still be waiting for processing. Am I wrong?

I think that oldestXidHavingUndo should be advanced at the time transaction
commits or when the undo log of an aborted transaction has been applied. Then
the discard worker would simply discard the undo log up to
oldestXidHavingUndo. However, as the transactions whose undo is still not
applied may no longer be registered in the shared memory (proc array), I don't
know how to determine the next value of oldestXidHavingUndo.

Also I wonder if FullTransactionId is needed for oldestXidHavingUndo in the
shared memory rather than plain TransactionId (see
oldestXidWithEpochHavingUndo in PROC_HDR). I think that the value cannot lag
behind nextFullXid by more than 2 billions transactions anyway because in that
case it would cause XID wraparound. (That in turn makes me think that VACUUM
FREEZE should be able to discard undo log too.)

[1] https://github.com/EnterpriseDB/zheap/tree/master

--
Antonin Houska
Web: https://www.cybertec-postgresql.com