Re: BUG #14206: Switch to using POSIX semaphores on FreeBSD

Tom, my diagnosis may be a nonsense, but the crash is real.

Jun  5 21:47:38 sippy postgres[3744]: [2-1] PANIC:  semop(id=65608) failed:
Invalid argument
Jun  5 21:47:38 sippy postgres[3743]: [2-1] PANIC:  semop(id=65608) failed:
Invalid argument
Jun  5 21:47:39 sippy postgres[3725]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:39 sippy postgres[3742]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3550]: [2-1] PANIC:  semop(id=65611) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3664]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3667]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3663]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3666]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 21:47:42 sippy postgres[3665]: [2-1] PANIC:  semop(id=65609) failed:
Invalid argument
Jun  5 22:30:56 sippy postgres[3632]: [2-1] PANIC:  semop(id=65610) failed:
Invalid argument
Jun  5 22:30:56 sippy postgres[3633]: [2-1] PANIC:  semop(id=65610) failed:
Invalid argument

From the man semopt:

     [EINVAL]           No semaphore set corresponds to semid, or the
process
                        would exceed the system-defined limit for the number
                        of per-process SEM_UNDO structures.

AFAIK we've been hitting the second cause there. This is with FreeBSD 10.3
and postgresql 9.2.16 specifically. We've also seen this with 9.1 I think,
but I am not 100% sure. The specific limit that got exceeded
was kern.ipc.semmnu.

I can probably dig some stack traces, although they might be cleared out
now. Now looking at it again, I think you is probably right this is not
happening on the primitive creation time, but during its actual use. Which
may make graceful handling tricky if possible at all, still "good to have"
from my PG user's point of view.

Nevertheless, it just makes using POSIX primitives even more attractive
IMHO.

On Wed, Jun 22, 2016 at 8:15 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Maxim Sobolev <sobomax@freebsd.org> writes:
> > Tom, on the related note on merits of SYSV semaphores vs. POSIX the
> > handling of SYSV semaphore shortage in PG is, uggh, awful. The whole
> server
> > crashes (abort()s), bumping into that on production box is not fun at all
> > and it's particularly easy since the resource is severely constrained by
> > default. Ideally it should just deny the particular connection request.
>
> This seems like nonsense, because those are acquired once at postmaster
> startup, not per connection.  You will need to decrease max_connections
> to start successfully in a resource-constrained system, but the same is
> true of other resource limits.
>
>                         regards, tom lane
>
>


--
Maksym Sobolyev
Sippy Software, Inc.
Internet Telephony (VoIP) Experts
Tel (Canada): +1-778-783-0474
Tel (Toll-Free): +1-855-747-7779
Fax: +1-866-857-6942
Web: http://www.sippysoft.com
MSN: sales@sippysoft.com
Skype: SippySoft