Обсуждение: PostgreSQL's share_buffer calculation using shmget() versus kernel.shmmax

On Thu, Feb 7, 2013 at 6:28 AM, Mel Llaguno <mllaguno@coverity.com> wrote:
> All,
>
> I'm wondering about how postgresql calculates the value for shared buffers
> as I see some discrepancies with what the following script provides versus
> what is recommended in the pgctl.log when the database fails to start.
>
> #!/bin/bash
> # simple shmsetup script
> page_size=`getconf PAGE_SIZE`
> phys_pages=`getconf _PHYS_PAGES`
> shmall=`expr $phys_pages / 2`
> shmmax=`expr $shmall \* $page_size`
> echo kernel.shmmax = $shmmax
> echo kernel.shmall = $shmall
>
> Any pointers which would explain these differences would be greatly
> appreciate. Ultimately, I'd like to calculate the expected kernel.shmmax
> which matches postgresql's shmget() call.
>

I don't know where you got hold of this script, but it seems this is
setting SHMMAX equal to half the size of RAM. So if your system has
4GB RAM, a process can request maximum of 2GB of shared memory. The
amount of shared memory PostgreSQL needs is governed by several
configuration parameters, but the most important and the one which
will eat up most of that shared memory is "shared_buffers". So if you
want to work within the bounds of SHMMAX set by this script, you would
need to set shared_buffers a notch lower than that. But often its
easier and better to decide your Postgres configuration parameters and
then set SHMMAX to satisfy that.

Thanks,
Pavan


--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

Pavan,

Thanks for your reply. I agree with your statement that you should set the configuration parameters first, but I would
liketo be able to calculate the SHMMAX value based on those parameters. This is particularly useful when suggesting
postgresql.confoptimizations to our customers whose machine have a lot of RAM (64+GB). Having to guess this value is
farfrom ideal; what I'd like is the formula used by postgresql that generates the shmget() value displayed in the
pgctl.log.

Thanks,

Mel
________________________________________
From: Pavan Deolasee [pavan.deolasee@gmail.com]
Sent: Wednesday, February 06, 2013 10:12 PM
To: Mel Llaguno
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] PostgreSQL's share_buffer calculation using shmget() versus kernel.shmmax

On Thu, Feb 7, 2013 at 6:28 AM, Mel Llaguno <mllaguno@coverity.com> wrote:
> All,
>
> I'm wondering about how postgresql calculates the value for shared buffers
> as I see some discrepancies with what the following script provides versus
> what is recommended in the pgctl.log when the database fails to start.
>
> #!/bin/bash
> # simple shmsetup script
> page_size=`getconf PAGE_SIZE`
> phys_pages=`getconf _PHYS_PAGES`
> shmall=`expr $phys_pages / 2`
> shmmax=`expr $shmall \* $page_size`
> echo kernel.shmmax = $shmmax
> echo kernel.shmall = $shmall
>
> Any pointers which would explain these differences would be greatly
> appreciate. Ultimately, I'd like to calculate the expected kernel.shmmax
> which matches postgresql's shmget() call.
>

I don't know where you got hold of this script, but it seems this is
setting SHMMAX equal to half the size of RAM. So if your system has
4GB RAM, a process can request maximum of 2GB of shared memory. The
amount of shared memory PostgreSQL needs is governed by several
configuration parameters, but the most important and the one which
will eat up most of that shared memory is "shared_buffers". So if you
want to work within the bounds of SHMMAX set by this script, you would
need to set shared_buffers a notch lower than that. But often its
easier and better to decide your Postgres configuration parameters and
then set SHMMAX to satisfy that.

Thanks,
Pavan


--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

On Thu, Feb 7, 2013 at 11:01 AM, Mel Llaguno <mllaguno@coverity.com> wrote:
> Having to guess this value is far from ideal; what I'd like is the formula used by postgresql that generates the
shmget()value displayed in the pgctl.log. 
>

There is no easy way or at least none that I'm aware of, to get the
exact value of shared memory needed for Postgres. If you have access
to the source code, you can look at CreateSharedMemoryAndSemaphores()
function in src/backend/storage/ipc/ipci.c to see what all goes in
determining the size for shmget()

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

Pavan,

Thanks. I'll have a look at the source code.

M.
________________________________________
From: Pavan Deolasee [pavan.deolasee@gmail.com]
Sent: Wednesday, February 06, 2013 10:41 PM
To: Mel Llaguno
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] PostgreSQL's share_buffer calculation using shmget() versus kernel.shmmax

On Thu, Feb 7, 2013 at 11:01 AM, Mel Llaguno <mllaguno@coverity.com> wrote:
> Having to guess this value is far from ideal; what I'd like is the formula used by postgresql that generates the
shmget()value displayed in the pgctl.log. 
>

There is no easy way or at least none that I'm aware of, to get the
exact value of shared memory needed for Postgres. If you have access
to the source code, you can look at CreateSharedMemoryAndSemaphores()
function in src/backend/storage/ipc/ipci.c to see what all goes in
determining the size for shmget()

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

Mel Llaguno <mllaguno@coverity.com> writes:
> Thanks for your reply. I agree with your statement that you should set the configuration parameters first, but I
wouldlike to be able to calculate the SHMMAX value based on those parameters. This is particularly useful when
suggestingpostgresql.conf optimizations to our customers whose machine have a lot of RAM (64+GB). Having to guess this
valueis far from ideal; what I'd like is the formula used by postgresql that generates the shmget() value displayed in
thepgctl.log. 

There's some rather old information in Table 17-2 here:
http://www.postgresql.org/docs/9.2/static/kernel-resources.html#SYSVIPC

As Pavan says, the shared_buffers term is usually the only one worth
worrying about.  The traditional advice is to not set that to more than
about a quarter of your physical RAM, which would mean that this script
you're using to set SHMMAX is leaving lots of headroom, which is
perfectly OK.  (AFAIK there is no penalty to setting SHMMAX larger than
you need.)

There's more info worth looking at here:
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

            regards, tom lane

Tom,

Thanks for the response. I've been doing a lot of performance tuning for our customers and I've found that wiki link a
lifesaver ;-) 

I'm trying to come up with a precise way to calculate the shmget() value which postgresql uses in the pgctl.log message
whenthe kernel.shmmax is set too low. There are situations when knowing this exact value is useful as our customers are
sometimesnot as familiar with postgresql as we'd like. Being able to calculate this value from enabled settings in
postgresql.confwould help us provide accurate guidance. As per Pavan's suggestion, I'm having a look at the
src/backend/storage/ipc/ipci.c.

Thanks,

Mel
________________________________________
From: Tom Lane [tgl@sss.pgh.pa.us]
Sent: Wednesday, February 06, 2013 10:49 PM
To: Mel Llaguno
Cc: Pavan Deolasee; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] PostgreSQL's share_buffer calculation using shmget() versus kernel.shmmax

Mel Llaguno <mllaguno@coverity.com> writes:
> Thanks for your reply. I agree with your statement that you should set the configuration parameters first, but I
wouldlike to be able to calculate the SHMMAX value based on those parameters. This is particularly useful when
suggestingpostgresql.conf optimizations to our customers whose machine have a lot of RAM (64+GB). Having to guess this
valueis far from ideal; what I'd like is the formula used by postgresql that generates the shmget() value displayed in
thepgctl.log. 

There's some rather old information in Table 17-2 here:
http://www.postgresql.org/docs/9.2/static/kernel-resources.html#SYSVIPC

As Pavan says, the shared_buffers term is usually the only one worth
worrying about.  The traditional advice is to not set that to more than
about a quarter of your physical RAM, which would mean that this script
you're using to set SHMMAX is leaving lots of headroom, which is
perfectly OK.  (AFAIK there is no penalty to setting SHMMAX larger than
you need.)

There's more info worth looking at here:
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

                        regards, tom lane

On Feb 6, 2013, at 5:58 PM, Mel Llaguno wrote:

> Any pointers which would explain these differences would be greatly appreciate.

Postgres is likely not the only thing on your system that allocates shared memory.

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice