On Tue, 9 Apr 2024 at 13:12, 范润泽 <davidfrz1996@126.com> wrote:
>
> Hi, Hackers
> I have observed that there has been a paucity of discussion concerning the parallel replay of WAL logs.
> The most recent discourse on this subject was during the PGCon event in 2023, where it was noted that PostgreSQL
utilizesa single process for WAL replay.
> However, when configuring primary-secondary replication, there exists a tangible scenario where the primary
accumulatesan excessive backlog that the secondary cannot replay promptly.
> This situation prompts me to question whether it is pertinent to consider integrating a parallel replay feature.
> Such a functionality could potentially mitigate the risk of delayed WAL application on replicas and enhance overall
systemresilience and performance.
> I am keen to hear your thoughts on this issue and whether you share the view that parallel WAL replay is a necessity
thatwe should collaboratively explore further.
I think we should definitely explore this further, yes.
Note that parallel WAL replay is not the only thing we can improve
here: A good part of WAL redo time is spent in reading and validating
the WAL records. If we can move that part to a parallel worker/thread,
that could probably improve the throughput by a good margin.
Then there is another issue with parallel recovery that I also called
out at PGCon 2023: You can't just reorder WAL records in a simple
page- or relation-based parallelization approach.
Indexes often contain transient information, of which replay isn't
easily parallelizable with other relation's redo due to their
dependencies on other relation's pages while gauranteeing correct
query results.
E.g., the index insertion of page 2 in one backend's operations order
{ 1) Page 1: heap insertion, 2) Page 2: index insert, 3) commit }
cannot be reordered to before the heap page insertion at 1, because
that'd allow a concurrent index access after replay of 2), but before
replay of 1), to see an "all-visible" page 1 in its index scan, while
the heap tuple of the index entry wasn't even inserted yet. Index-only
scans could thus return invalid results.
See also the wiki page [0] on parallel recovery, and Koichi-san's
repository [1] with his code for parallel replay based on PG 14.6.
Kind regards,
Matthias van de Meent
[0] https://wiki.postgresql.org/wiki/Parallel_Recovery
[1] https://github.com/koichi-szk/postgres/commits/parallel_replay_14_6/