Re: [PATCH] Accept connections post recovery without waiting for RemoveOldXlogFiles

On Mon, Sep 8, 2025 at 3:03 PM Nitin Motiani <nitinmotiani@google.com> wrote:
>
> Hi Hackers,
>
> I'd like to propose a patch to allow accepting connections post recovery without waiting for the removal of old xlog
files.
>
> Why : We have seen instances where the crash recovery takes very long (tens of minutes to hours) if a large number of
accumulatedWAL files need to be cleaned up (eg : Cleaning up 2M old WAL files took close to 4 hours). 
>
> This WAL accumulation is usually caused by :
>
> 1. Inactive replication slot
> 2. PITR failing to keep up
>
> In the above cases when the resolution (deleting inactive slot/disabling PITR) is followed by a crash (before
checkpointcould run), we see the recovery take a very long time. Note that in these cases the actual WAL replay is done
relativelyquickly and most of the delay is due to RemoveOldXlogFiles(). 

It makes sense to improve this.

> How : This patch solves this issue by running RemoveOldXlogFiles() separately and async. This is achieved by doing
twothings : 
>
> 1. Skip RemoveOldXlogFiles() for an END_OF_RECOVERY checkpoint. This will ensure that the recovery finishes sooner
andpostgres can start accepting connections. 
> 2. After the recovery we run another checkpoint without CHECKPOINT_WAIT. This is done in StartupXLOG(). This will
leadto some extra work but that should be minuscule as it is run right after the recovery. And the majority of work
doneby this checkpoint will be in RemoveOldXlogFiles() which can now run asynchronously. 
>
> I considered a couple of alternative solutions before attempting this.
>
> 1. One option could be to simply skip the removal of old xlog files during recovery and let a later checkpoint take
careof that. But in case of large checkpoint_timeout, this could lead to bloat for longer. 
>
> 2. Another approach might be to separate out RemoveOldXlogFiles() in a new request. This might also be doable by
creatinga special checkpoint flag like CHECKPOINT_ONLY_DELETE_OLD_FILES and using that in RequestCheckpoint(). This way
wecan have the second checkpoint only take care of file deletion. I ended up picking my approach over this because that
canbe done with a smaller change which might make it safer and less error-prone. 

One of the advantages of this approach over forcing an extra
checkpoint is that you don't need to loop through the entire buffer
pool just to find out mostly nothing is dirty, but yeah this may
create some extra flags and extra checks in checkpointer code.

--
Regards,
Dilip Kumar
Google