On Thu, Dec 10, 2015 at 5:13 PM, Carlo Cabanilla <carlo@datadoghq.com> wrote:
> I'm trying to figure out why we had a build up of connections on
> our streaming replica.
Seriously, from the data provided, about all I can say is "because
you were opening them faster than you were closing them". You
don't say how many cores or how much RAM you had, but allowing 2000
connections is rarely a good idea.
http://tbeitr.blogspot.com/2015/11/for-better-service-please-take-number.html
> We're running postgres 9.3.5 on the master and 9.3.10 on
> the replica,
Because of data loss bugs in 9.3.5 which are fixed in 9.3.10, I
recommend you update the master ASAP.
> linux 3.2.0 on both,
There are scheduler problems in kernels at that level; I recommend
upgrading linux to a 3.8 or later kernel; 3.13 or later is best, if
possible.
I don't suppose you have vmstat 1 output from the incident? If it
happens again, try to capture that.
Have you tuned the OS vm.dirty_* settings? How about the
transparent_hugepage settings?
If this is a NUMA environment, what does this show?:
numactl --hardware
--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company