Обсуждение: Fresh build on OS X not working (memory)

Gavin Kistner <phrogz@mac.com> writes:
> Mac OS X 10.6.3
> Built and installed PG 8.4.3 from source
> Trying to start the server, I get "could not create shared memory segment: Invalid argument".

OSX is *notoriously* unfriendly about configuring shm parameters.
You look at it sidewise, it doesn't work, and generally won't tell
you why it's refusing to talk to you.  We document a few of the known
constraints in our manual, but I'm sure there are more.

> Slim2:pgsql _postgres$ sysctl -a | grep shm
> kern.sysv.shmmax: 2147483648
> kern.sysv.shmmin: 1
> kern.sysv.shmmni: 32
> kern.sysv.shmseg: 8
> kern.sysv.shmall: 2147483648

Hm.  That shmall setting is not sane at all (remember the units are
pages).  I wonder whether you've forced some internal overflow inside
the kernel.  For comparison, I have these settings on my laptop:

pro:~ tgl$ cat /etc/sysctl.conf
kern.sysv.shmmax=33554432
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=8192

and PG starts fine for me.

            regards, tom lane

On Apr 29, 2010, at 11:42 PM, Tom Lane wrote:
> For comparison, I have these settings on my laptop:
>
> pro:~ tgl$ cat /etc/sysctl.conf
> kern.sysv.shmmax=33554432
> kern.sysv.shmmin=1
> kern.sysv.shmmni=32
> kern.sysv.shmseg=8
> kern.sysv.shmall=8192
>
> and PG starts fine for me.

I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks!

The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about
thefollowing settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*? 

kern.sysv.shmmax=1610612736
kern.sysv.shmall=393216
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
kern.maxproc=2048

Gavin Kistner <phrogz@mac.com> writes:
> On Apr 29, 2010, at 11:42 PM, Tom Lane wrote:
>> For comparison, I have these settings on my laptop:
>> pro:~ tgl$ cat /etc/sysctl.conf
>> kern.sysv.shmmax=33554432
>> kern.sysv.shmmin=1
>> kern.sysv.shmmni=32
>> kern.sysv.shmseg=8
>> kern.sysv.shmall=8192
>>
>> and PG starts fine for me.

> I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks!

> The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about
thefollowing settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*? 

> kern.sysv.shmmax=1610612736
> kern.sysv.shmall=393216
> kern.sysv.shmmin=1
> kern.sysv.shmmni=32
> kern.sysv.shmseg=8
> kern.maxprocperuid=512
> kern.maxproc=2048

I have no idea why EDB recommend changing the maxproc settings, but
I doubt that's related to shared memory.  I see that they have shmmax
equal to exactly 4096 times shmall, so that's good, but there must be
some other OSX peculiarity that this is tripping over.  Maybe it's too
large ... do you actually have 1.6GB of RAM available that could
reasonably be dedicated to PG shared memory?  Another possibility is
that maybe only exact powers of 2 work well.  I'm just guessing though.

            regards, tom lane

I wrote:
> I have no idea why EDB recommend changing the maxproc settings, but
> I doubt that's related to shared memory.  I see that they have shmmax
> equal to exactly 4096 times shmall, so that's good, but there must be
> some other OSX peculiarity that this is tripping over.  Maybe it's too
> large ... do you actually have 1.6GB of RAM available that could
> reasonably be dedicated to PG shared memory?  Another possibility is
> that maybe only exact powers of 2 work well.  I'm just guessing though.

I poked around in the OS X kernel sources (let's hear it for open
source) and found that there really isn't a lot of special magic around
the shm control parameters anymore: as of OS X 10.6, they're all 64 bits
and the comparisons are entirely straightforward.  If you can get the
values into the system, which you did since sysctl was reporting them,
they should work as-expected.

So why were you getting an EINVAL failure?  The only theory that seems
to hold water after looking at the kernel source code is that there was a
pre-existing shm segment with the same key, and you got to this bit in
shmget_existing():

        if (uap->size && uap->size > shmseg->u.shm_segsz)
                return EINVAL;

       if ((uap->shmflg & (IPC_CREAT | IPC_EXCL)) == (IPC_CREAT | IPC_EXCL))
                return EEXIST;

IOW, if there's an existing shm segment of the same key but it's too
small, you get EINVAL.  Not EEXIST, which is what our code in
InternalIpcMemoryCreate is expecting to see for a collision.  So our
code fails to recognize the case as a collision and spits out a
misleading error message.

If memory serves, we've seen this issue before [ ... digs in archives
... ] ah-hah:

http://archives.postgresql.org/pgsql-hackers/2008-10/msg00896.php

I argued in that message that returning EINVAL when EEXIST would apply
is a kernel bug, and I still think that.  But what seems likely at this
point is that many BSD-derived kernels behave this way, and we're never
gonna get 'em all fixed.  So it would behoove us to work around it.

Does the theory of a pre-existing smaller shmem segment make sense from
your end?  In particular, had you previously had another Postgres server
running on that machine, and perhaps killed it ungracefully?  If this
theory is correct, the issue was "fixed" as a result of rebooting (thus
making the old segment go away), not as a result of any changes of the
shm parameters.  OTOH I'd have expected you to have to reboot multiple
times while experimenting with the shm parameters, so I'm not entirely
convinced I've hit on the right explanation.

            regards, tom lane

I wrote:
> ... OTOH I'd have expected you to have to reboot multiple
> times while experimenting with the shm parameters, so I'm not entirely
> convinced I've hit on the right explanation.

Oh ... were you using sysctl -w rather than rebooting to adjust the
shmmem parameters?  I realized after experimenting that 10.6 lets you
do that.  It didn't use to be possible to change these parameters in
OS X without rebooting --- I wonder when they fixed that.

            regards, tom lane

On Apr 30, 2010, at 3:35 PM, Tom Lane wrote:
> Does the theory of a pre-existing smaller shmem segment make sense from
> your end?  In particular, had you previously had another Postgres server
> running on that machine, and perhaps killed it ungracefully?  If this
> theory is correct, the issue was "fixed" as a result of rebooting (thus
> making the old segment go away), not as a result of any changes of the
> shm parameters.  OTOH I'd have expected you to have to reboot multiple
> times while experimenting with the shm parameters, so I'm not entirely
> convinced I've hit on the right explanation.

It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was
workingwith the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a
manualuninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed
asdescribed earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a
reboot.The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well. 

Nice sleuthing. If I didn't have so many things going on at the moment I'd revert to the supplied settings and test to
ensureit works and provide a confirmation. I'll try to remember to do that later. 

Thanks again for your help and good information.

Gavin Kistner <phrogz@mac.com> writes:
> On Apr 30, 2010, at 3:35 PM, Tom Lane wrote:
>> Does the theory of a pre-existing smaller shmem segment make sense from
>> your end?  In particular, had you previously had another Postgres server
>> running on that machine, and perhaps killed it ungracefully?

> It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was
workingwith the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a
manualuninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed
asdescribed earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a
reboot.The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well. 

Yeah, I was able to reproduce the problem locally by kill -9'ing the
postmaster and then enlarging max_connections in postgresql.conf before
trying to restart.  In your case it might've been caused by the new
version needing a bit more shared memory rather than by any change in
parameter settings, but the basic issue seems pretty clear now.

I've applied a patch to our CVS to make PG behave more sanely in this
case.  It does do the right thing if it recognizes that the shmget
failure is due to a pre-existing segment, but we weren't being
sufficiently careful about distinguishing that from a bad size request.

Thanks for the report!

            regards, tom lane

On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Gavin Kistner <phrogz@mac.com> writes:
>> Mac OS X 10.6.3
>> Built and installed PG 8.4.3 from source
>> Trying to start the server, I get "could not create shared memory segment: Invalid argument".
>
> OSX is *notoriously* unfriendly about configuring shm parameters.
> You look at it sidewise, it doesn't work, and generally won't tell
> you why it's refusing to talk to you.  We document a few of the known
> constraints in our manual, but I'm sure there are more.

The fact that they seem to change the needed magical incantations
every point release doesn't help things either.

Scott Marlowe <scott.marlowe@gmail.com> writes:
> On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> OSX is *notoriously* unfriendly about configuring shm parameters.
>> You look at it sidewise, it doesn't work, and generally won't tell
>> you why it's refusing to talk to you. �We document a few of the known
>> constraints in our manual, but I'm sure there are more.

> The fact that they seem to change the needed magical incantations
> every point release doesn't help things either.

Actually, it appears they've finally reached a point of reasonable
sanity in 10.6.  It's still true that they reject non-page-multiple
SHMMAX, instead of rounding to the next page boundary like everybody
else does; but the old kluges whereby you could only set the values by
waving a dead chicken in a specific fashion during boot seem to be
mostly gone.  I wasn't aware of this myself, not having experimented
recently.

            regards, tom lane