Re: Scaling shared buffer eviction

On 9/16/14, 8:18 AM, Amit Kapila wrote:

I think the main reason for slight difference is that
when the size of shared buffers is almost same as data size, the number
of buffers it needs from clock sweep are very less, as an example in first
case (when size of shared buffers is 12286MB), it actually needs at most
256 additional buffers (2MB) via clock sweep, where as bgreclaimer
will put 2000 (high water mark) additional buffers (0.5% of shared buffers
is greater than 2000 ) in free list, so bgreclaimer does some extra work
when it is not required

This is exactly what I was warning about, as the sort of lesson learned from the last round of such tuning.  There are going to be spots where trying to tune the code to be aggressive on the hard cases will work great.  But you need to make that dynamic to some degree, such that the code doesn't waste a lot of time sweeping buffers when the demand for them is actually weak.  That will make all sorts of cases that look like this slower.

We should be able to tell these apart if there's enough instrumentation and solid logic inside of the program itself though.  The 8.3 era BGW coped with a lot of these issues using a particular style of moving average with fast reaction time, plus instrumenting the buffer allocation rate as accurately as it could. So before getting into high/low water note questions, are you comfortable that there's a clear, accurate number that measures the activity level that's important here?

And have you considered ways it might be averaging over time or have a history that's analyzed?

The exact fast approach / slow decay weighted moving average approach of the 8.3 BGW, the thing that tried to smooth the erratic data set possible here, was a pretty critical part of getting itself auto-tuning to workload size.  It ended up being much more important than the work of setting the arbitrary watermark levels.