Обсуждение: Hit by the out of memory killer last night
Last night we were hit by the out of memory killer. Looking at the following graph, you can clearly see unusual memory growth. This is a database server running Postgres 9.0.0.
http://mediafly-public.s3.amazonaws.com/dbcluster02-master-month.png
We have another server, running Postgres 9.0.1 which showing a similar behavior pattern:
http://mediafly-public.s3.amazonaws.com/dbcluster03-master-month.png
Compare these to another server which is running 8.3.7 and does not exhibit this behavior:
http://mediafly-public.s3.amazonaws.com/dbcluster01-master-month.png
We're going to accelerate our upgrade to 9.0.2 for these servers, however, I'm concerned that we have not identified the source of the memory leak and this upgrade won't necessarily fix the problem.
Any advice? What should I be looking for?
Thanks,
Bryan
http://mediafly-public.s3.amazonaws.com/dbcluster02-master-month.png
We have another server, running Postgres 9.0.1 which showing a similar behavior pattern:
http://mediafly-public.s3.amazonaws.com/dbcluster03-master-month.png
Compare these to another server which is running 8.3.7 and does not exhibit this behavior:
http://mediafly-public.s3.amazonaws.com/dbcluster01-master-month.png
We're going to accelerate our upgrade to 9.0.2 for these servers, however, I'm concerned that we have not identified the source of the memory leak and this upgrade won't necessarily fix the problem.
Any advice? What should I be looking for?
Thanks,
Bryan
On Mon, Jan 31, 2011 at 10:35 AM, Ben Chobot <bench@silentmedia.com> wrote:
Simply an oversight. That being said, it does not explain WHY we are seeing such a memory usage pattern.
Thanks,
Bryan
> Any advice? What should I be looking for?Any particular reason you are running the OOM killer on a database server? Why have the kernel set to overcommit memory in the first place?
Simply an oversight. That being said, it does not explain WHY we are seeing such a memory usage pattern.
Thanks,
Bryan
On Jan 31, 2011, at 7:55 AM, Bryan Murphy wrote: > Last night we were hit by the out of memory killer. Looking at the following graph, you can clearly see unusual memorygrowth. This is a database server running Postgres 9.0.0. [snip] > Any advice? What should I be looking for? Any particular reason you are running the OOM killer on a database server? Why have the kernel set to overcommit memory inthe first place?