I guess details on io waits/ iops saturation etc metrics would need to be ruled out for further discussion.
Do the dashboards wrt above metrics look ok?
We had vms with unlimited iops and no synchronous replication, so I do not recall this piling up of locks issues atleast till 1tb dbs on ssds till pg11.
Googling does mention some things wrt tuning of wal buffers etc, but I believe ruling out resource exhaustion is important.
My gues is that you have too many active client connections, and you are suffering
from contention between the many backends that all want to write WAL.
In that case, use a connection pool to limit the number of active connections.
We do have pgbouncer in place already.
Thanks for the replies so far.
What I really want to know in this case is if there is some other PG operation that accounts for a WALWriteLock wait, or is it always an I/O (write) to the WAL file storage, and we can focus our investigation there?