Обсуждение: shared_buffers vs Linux file cache

Поиск
Список
Период
Сортировка

shared_buffers vs Linux file cache

От
Huan Ruan
Дата:
Hi All

I thought 'shared_buffers' sets how much memory that is dedicated to PostgreSQL to use for caching data, therefore not available to other applications.

However, as shown in the following screenshots, The server (CentOS 6.6 64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the free+buffer+cache is 60GB. 

Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
Is 'shared_buffers' pre allocated to Postgres, and Postgres only?

Thanks
Huan


Inline images 2

Inline images 1
Вложения

Re: shared_buffers vs Linux file cache

От
Glyn Astill
Дата:
> From: Huan Ruan <huan.ruan.it@gmail.com>
>To: pgsql-performance@postgresql.org
>Sent: Thursday, 15 January 2015, 11:30
>Subject: [PERFORM] shared_buffers vs Linux file cache
>
>
>
>Hi All
>
>
>I thought 'shared_buffers' sets how much memory that is dedicated to PostgreSQL to use for caching data, therefore not
availableto other applications. 
>
>
>However, as shown in the following screenshots, The server (CentOS 6.6 64bit) has 64GB of RAM, and 'shared_buffer' is
setto 32GB, but the free+buffer+cache is 60GB.  
>
>
>Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
>Is 'shared_buffers' pre allocated to Postgres, and Postgres only?

>

I've not looked at the images, but I think you're getting PostgreSQL shared_buffers and the OS buffercache mixed up;
theyare not the same. 

PostgreSQL shared_buffers is specific to postgres, whereas the OS buffercache will just use free memory to cache data
pagesfrom disk, and this is what you're seeing. 

Some reading for you:
http://www.tldp.org/LDP/sag/html/buffer-cache.html

Glyn


Re: shared_buffers vs Linux file cache

От
Jeff Janes
Дата:

On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
Hi All

I thought 'shared_buffers' sets how much memory that is dedicated to PostgreSQL to use for caching data, therefore not available to other applications.

However, as shown in the following screenshots, The server (CentOS 6.6 64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the free+buffer+cache is 60GB. 

Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
Is 'shared_buffers' pre allocated to Postgres, and Postgres only?

While PostgreSQL has reserves the right to use 32GB, as long as PostgreSQL has not actually dirtied that RAM yet, then the kernel is free to keep using it to cache files.

 
Cheers,

Jeff

Re: shared_buffers vs Linux file cache

От
Tom Lane
Дата:
Jeff Janes <jeff.janes@gmail.com> writes:
> On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
>> I thought 'shared_buffers' sets how much memory that is dedicated to
>> PostgreSQL to use for caching data, therefore not available to other
>> applications.

> While PostgreSQL has reserves the right to use 32GB, as long as PostgreSQL
> has not actually dirtied that RAM yet, then the kernel is free to keep
> using it to cache files.

Another thing to keep in mind is that, even if Postgres *has* used the
RAM, the kernel might decide to swap parts of it out if it's not being
used heavily.  This is pretty disastrous from a performance standpoint,
so it's advisable to not make shared_buffers very much larger than what
your application will keep "hot".

Ideally we'd lock the shared buffer arena into RAM to prevent that,
but such facilities are often unavailable or restricted to root.

            regards, tom lane


Re: shared_buffers vs Linux file cache

От
Huan Ruan
Дата:
Thanks very much, Glyn, Jeff, and Tom. That was very clearly explained.

A related case, see the following top dump. The Postgres process is using 87g residential memory, which I thought was the physical memory consumed by a process that can't be shared with others. While, the free+cached is about 155gb. But, (87 + 155) is bigger than the total available 198g RAM. Does this mean some of the residential memory used by Postgres is actually shareable to others?
   

>> Mem:  198311880k total, 183836408k used, 14475472k free, 8388k buffers
>> Swap:  4194300k total,   314284k used,  3880016k free, 141105408k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
>> 15338 postgres  20   0 97.9g  87g  87g S  0.3 46.4  21:47.44 
>> postgres: checkpointer process
>> 27473 postgres  20   0 98.1g  29g  29g S  0.0 15.8   2:14.93 
>> postgres: xxxx idle
>>  4710 postgres  20   0 98.1g  24g  23g S  0.0 12.7   1:17.41 
>> postgres: xxxx idle 
>> 26587 postgres 20 0 98.0g 15g 15g S 0.0 8.0 1:21.24 
 


 

Re: shared_buffers vs Linux file cache

От
Roman Konoval
Дата:
Huan,

Residential memory is part of the process memory that is now swapped and is in RAM. This includes also memory shared with other processes so sum of RES for all processes may be greater that total physical memory.

I recommend this article http://www.depesz.com/2012/06/09/how-much-ram-is-postgresql-using/ to get better understanding how linux manages memory.

Regards,
Roman

On Fri, Jan 16, 2015 at 5:32 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
Thanks very much, Glyn, Jeff, and Tom. That was very clearly explained.

A related case, see the following top dump. The Postgres process is using 87g residential memory, which I thought was the physical memory consumed by a process that can't be shared with others. While, the free+cached is about 155gb. But, (87 + 155) is bigger than the total available 198g RAM. Does this mean some of the residential memory used by Postgres is actually shareable to others?
   

>> Mem:  198311880k total, 183836408k used, 14475472k free, 8388k buffers
>> Swap:  4194300k total,   314284k used,  3880016k free, 141105408k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
>> 15338 postgres  20   0 97.9g  87g  87g S  0.3 46.4  21:47.44 
>> postgres: checkpointer process
>> 27473 postgres  20   0 98.1g  29g  29g S  0.0 15.8   2:14.93 
>> postgres: xxxx idle
>>  4710 postgres  20   0 98.1g  24g  23g S  0.0 12.7   1:17.41 
>> postgres: xxxx idle 
>> 26587 postgres 20 0 98.0g 15g 15g S 0.0 8.0 1:21.24