What is the latency between the two sites? That is going to dictate how much potential throughput a given TCP/IP stream could use. You may need tune your TCP/IP window to increase the throughput.
Ping shows a pretty solid latency of 49-50 ms.
47 packets transmitted, 47 received, 0% packet loss, time 46057ms
rtt min/avg/max/mdev = 49.125/49.260/49.682/0.289 ms
I would check the network to ensure there are no errors occurring; i.e. dropped packets and retransmits.
Looking at ifconfig, I do see dropped RX packets on the iface that talks to the replica. I don't see any dropped packets on the replica iface that talks to the primary. The local LAN iface for both also has many dropped packets.
Do you have QOS enabled and if so how is that handled? Does it just drop the packets? As that would cause the TCP/IP to backoff and retransmit.
I'll follow-up with my sys admin team here. I believe there is QoS enabled but don't have details yet.