Обсуждение: SHMMAX and shared_bufffers
Hi, I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) and my shared_buffers value in postgresql.conf to 256MB the server fails to start. I managed to find a tipping point: 249MB seems to be too much and 248MB seems to be okay. Could someone explain what I'm missing here? I thought the shared_buffers could be anything up to SHMMAX? Thanks in advance Thom
Actually, I think I may have solved it, but I would like someone to verify it. The temp_buffers takes up 8MB which suggests that needs to be taken into account, so 256MB - 8MB = 248MB. Is this right? Thanks Thom On Tue, Nov 11, 2008 at 4:57 PM, Thom Brown <thombrown@gmail.com> wrote: > Hi, > > I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) > and my shared_buffers value in postgresql.conf to 256MB the server > fails to start. I managed to find a tipping point: 249MB seems to be > too much and 248MB seems to be okay. > > Could someone explain what I'm missing here? I thought the > shared_buffers could be anything up to SHMMAX? > > Thanks in advance > > Thom >
Thom Brown escribió: > Hi, > > I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) > and my shared_buffers value in postgresql.conf to 256MB the server > fails to start. I managed to find a tipping point: 249MB seems to be > too much and 248MB seems to be okay. > > Could someone explain what I'm missing here? I thought the > shared_buffers could be anything up to SHMMAX? shared_buffers is not the only factor to shared memory, so you need to provide some extra SHMMAX slop. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Tue, Nov 11, 2008 at 02:17:20PM -0300, Alvaro Herrera wrote: > Thom Brown escribió: > > I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) > > and my shared_buffers value in postgresql.conf to 256MB the server > > fails to start. I managed to find a tipping point: 249MB seems to be > > too much and 248MB seems to be okay. > > > > Could someone explain what I'm missing here? I thought the > > shared_buffers could be anything up to SHMMAX? > > shared_buffers is not the only factor to shared memory, so you need to > provide some extra SHMMAX slop. what is this "extra slop" needed for? I'd always ignored it before and just made SHMMAX a bit bigger until PG stopped complaining and started. But computers are supposed to be deterministic; what's taking up the extra space? I expect this knowledge to be most useful to people like Greg Smith who's making a config auto-tuner, it would be bad if it said to set SHMMAX to xMB and it really needs to be set to (x+12)MB. I've just found a table[1] in the docs that attempts to explain what's going on, but I get 10.5MB of extra shared memory being used (default config values, expect for shared_buffers going up to 256MB). Sam [1] http://www.postgresql.org/docs/current/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS
On Tue, Nov 11, 2008 at 10:58 AM, Sam Mason <sam@samason.me.uk> wrote: > On Tue, Nov 11, 2008 at 02:17:20PM -0300, Alvaro Herrera wrote: >> Thom Brown escribió: >> > I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) >> > and my shared_buffers value in postgresql.conf to 256MB the server >> > fails to start. I managed to find a tipping point: 249MB seems to be >> > too much and 248MB seems to be okay. >> > >> > Could someone explain what I'm missing here? I thought the >> > shared_buffers could be anything up to SHMMAX? >> >> shared_buffers is not the only factor to shared memory, so you need to >> provide some extra SHMMAX slop. > > what is this "extra slop" needed for? I'd always ignored it before and > just made SHMMAX a bit bigger until PG stopped complaining and started. > But computers are supposed to be deterministic; what's taking up the > extra space? I expect this knowledge to be most useful to people like > Greg Smith who's making a config auto-tuner, it would be bad if it said > to set SHMMAX to xMB and it really needs to be set to (x+12)MB. > > I've just found a table[1] in the docs that attempts to explain what's > going on, but I get 10.5MB of extra shared memory being used (default > config values, expect for shared_buffers going up to 256MB). free space map and temp buffers. Not sure what else. Note that there could always be some other application using some small amount of shared memory. use ipcs to see it. Run it as root to see other user's / process's shared mem usage.
Scott Marlowe escribió: > On Tue, Nov 11, 2008 at 10:58 AM, Sam Mason <sam@samason.me.uk> wrote: > > what is this "extra slop" needed for? > free space map and temp buffers. Not sure what else. Lock space, pg_clog, pg_subtrans and pg_multixact buffers, FSM, and some other things. External modules can request additional shared memory too. We're not very strict about the accounting. (temp buffers do not use shared memory -- what would be the point?) > Note that there could always be some other application using some > small amount of shared memory. use ipcs to see it. Run it as root to > see other user's / process's shared mem usage. shmmax is per-process. It's shmall that's system-wide, but that's measured in pages and is rarely a problem. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Sam Mason <sam@samason.me.uk> writes: > On Tue, Nov 11, 2008 at 02:17:20PM -0300, Alvaro Herrera wrote: >> shared_buffers is not the only factor to shared memory, so you need to >> provide some extra SHMMAX slop. > what is this "extra slop" needed for? See table 17-2 here http://www.postgresql.org/docs/8.3/static/kernel-resources.html#SYSVIPC regards, tom lane
On Tue, 11 Nov 2008, Thom Brown wrote: > I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) > and my shared_buffers value in postgresql.conf to 256MB the server > fails to start. I managed to find a tipping point: 249MB seems to be > too much and 248MB seems to be okay. Buffers are allocated in 8K blocks. When you use '256MB', that turns into 32768 of those. Each buffer has a header and some other overhead such that they actually take up 8400 bytes[1], which totals 1.025X the amount that actually goes into the buffer pool. There's another MB or two that goes into other overhead, that part depends on things like max_connections. That overhead computation isn't even exact in the source code![2] At shared_buffers=248MB, just the buffer pool overhead pushes the required amount to allocate up to 254.3MB; that just slips by with the rest of your settings. At 249MB, the buffer pool needs 255.3MB. With the other overhead added in, that just exceeds the 256MB SHMMAX and fails. 1: http://www.postgresql.org/docs/current/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS 2: http://doxygen.postgresql.org/ipci_8c.html#5371eff96f6dea948021ddfc9f0f5a38 -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Thanks guys. I think I've got a clearer idea of what's going on here. Basically I need to take more into account than just the shared buffers setting, and knowing that, I can leave extra head-room for other options which will likely take up shared memory. Cheers Thom On Wed, Nov 12, 2008 at 1:18 AM, Greg Smith <gsmith@gregsmith.com> wrote: > On Tue, 11 Nov 2008, Thom Brown wrote: > >> I've noticed that if I set my SHMMAX to 256 * 1024 * 1024 (268435456) >> and my shared_buffers value in postgresql.conf to 256MB the server >> fails to start. I managed to find a tipping point: 249MB seems to be >> too much and 248MB seems to be okay. > > Buffers are allocated in 8K blocks. When you use '256MB', that turns into > 32768 of those. Each buffer has a header and some other overhead such that > they actually take up 8400 bytes[1], which totals 1.025X the amount that > actually goes into the buffer pool. There's another MB or two that goes > into other overhead, that part depends on things like max_connections. That > overhead computation isn't even exact in the source code![2] > > At shared_buffers=248MB, just the buffer pool overhead pushes the required > amount to allocate up to 254.3MB; that just slips by with the rest of your > settings. At 249MB, the buffer pool needs 255.3MB. With the other overhead > added in, that just exceeds the 256MB SHMMAX and fails. > > 1: > http://www.postgresql.org/docs/current/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS > 2: > http://doxygen.postgresql.org/ipci_8c.html#5371eff96f6dea948021ddfc9f0f5a38 > > -- > * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD >