My gues is that you have too many active client connections, and you are suffering from contention between the many backends that all want to write WAL.
In that case, use a connection pool to limit the number of active connections.
We do have pgbouncer in place already.
Thanks for the replies so far.
What I really want to know in this case is if there is some other PG operation that accounts for a WALWriteLock wait, or is it always an I/O (write) to the WAL file storage, and we can focus our investigation there?