Обсуждение: Incorrect formula for SysV IPC parameters
Hello, I found that the formulas to calculate SEMMNI and SEMMNS are incorrect in 9.2 and later. http://www.postgresql.org/docs/9.5/static/kernel-resources.html All of them say that the same thing as following, | SEMMNI Maximum number of semaphore identifiers (i.e., sets) | | at least ceil((max_connections + autovacuum_max_workers + 4) / 16) | | SEMMNS Maximum number of semaphores system-wide | | ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 | plus room for other applications But actually the number of semaphores PostgreSQL needs is calculated as following in 9.4 and later. numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) MaxConnections = max_connections + autovacuum_max_workers + 1 + max_worker_processes So, the formula for SEMMNI should be ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) and SEMMNS should have the same fix. In 9.3 and 9.2, the documentation says the same thing but actually it is calculated as following, numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) MaxConnections = max_connections + autovacuum_max_workers + 1 + GetNumShmemAttachedBgworkers() Omitting GetNumShmemAttachedBgworkers, the actual formula is ceil((max_connections + autovacuum_max_workers + 5) / 16) In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. I attached two patches for 9.2-9.3 and 9.4-9.6dev respectively. patch command complains a bit on applying it on 9.2. On the platforforms that doesn't have tas operation needs additional 1024 + 64 semaphores but I understand it as out of scope of the documentation. One concern is that 'at least' and 'plus room for other applications' are mixed in the table 17-1, especially for SEMMNI and SEMMNS. It seems to me that they should be in the same wording, but it is not an actual problem. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote: > Hello, I found that the formulas to calculate SEMMNI and SEMMNS > are incorrect in 9.2 and later. > > http://www.postgresql.org/docs/9.5/static/kernel-resources.html > > All of them say that the same thing as following, > > | SEMMNI Maximum number of semaphore identifiers (i.e., sets) > | > | at least ceil((max_connections + autovacuum_max_workers + 4) / 16) > | > | SEMMNS Maximum number of semaphores system-wide > | > | ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 > | plus room for other applications > > But actually the number of semaphores PostgreSQL needs is > calculated as following in 9.4 and later. > > numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) > MaxConnections = max_connections + autovacuum_max_workers + 1 + > max_worker_processes > > So, the formula for SEMMNI should be > > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) > > and SEMMNS should have the same fix. > > > In 9.3 and 9.2, the documentation says the same thing but > actually it is calculated as following, > > numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) > MaxConnections = max_connections + autovacuum_max_workers + 1 + > GetNumShmemAttachedBgworkers() > > Omitting GetNumShmemAttachedBgworkers, the actual formula is > > ceil((max_connections + autovacuum_max_workers + 5) / 16) > > > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. > > > I attached two patches for 9.2-9.3 and 9.4-9.6dev > respectively. patch command complains a bit on applying it on > 9.2. Good catch! ISTM that you also need to change the descriptions about SEMMNI and SEMMNS under the Table 17-1. Regards, -- Fujii Masao
At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in <CAHGQGwHgBsat29_ZqK3aXg4a5Lsa0JUv579VkGWX3R_g0KOncw@mail.gmail.com> > On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI > <horiguchi.kyotaro@lab.ntt.co.jp> wrote: > > Hello, I found that the formulas to calculate SEMMNI and SEMMNS > > are incorrect in 9.2 and later. > > > > http://www.postgresql.org/docs/9.5/static/kernel-resources.html > > > > But actually the number of semaphores PostgreSQL needs is > > calculated as following in 9.4 and later. ... > > So, the formula for SEMMNI should be > > > > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) > > > > and SEMMNS should have the same fix. > > > > > > In 9.3 and 9.2, the documentation says the same thing but ... > > ceil((max_connections + autovacuum_max_workers + 5) / 16) > > > > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. > > Good catch! Thanks. > ISTM that you also need to change the descriptions about SEMMNI and SEMMNS > under the Table 17-1. Oops! Thank you for pointing it out. The original description doesn't mention the magic-number ('5' in the above formulas, which previously was '4') so I haven't added anything about it. Process of which the number is limited by max_worker_processes is called 'background process' (not 'backend worker') in the documentation so I used the name to mention it in the additional description. The difference in the body text for 9.2, 9.3 is only a literal '4' to '5' in the formula. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote: > At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in <CAHGQGwHgBsat29_ZqK3aXg4a5Lsa0JUv579VkGWX3R_g0KOncw@mail.gmail.com> >> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI >> <horiguchi.kyotaro@lab.ntt.co.jp> wrote: >> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS >> > are incorrect in 9.2 and later. >> > >> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html >> > >> > But actually the number of semaphores PostgreSQL needs is >> > calculated as following in 9.4 and later. > ... >> > So, the formula for SEMMNI should be >> > >> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) >> > >> > and SEMMNS should have the same fix. >> > >> > >> > In 9.3 and 9.2, the documentation says the same thing but > ... >> > ceil((max_connections + autovacuum_max_workers + 5) / 16) >> > >> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. >> >> Good catch! > > Thanks. > >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS >> under the Table 17-1. > > Oops! Thank you for pointing it out. > > The original description doesn't mention the magic-number ('5' in > the above formulas, which previously was '4') so I haven't added > anything about it. > > Process of which the number is limited by max_worker_processes is > called 'background process' (not 'backend worker') in the > documentation so I used the name to mention it in the additional > description. > > The difference in the body text for 9.2, 9.3 is only a literal > '4' to '5' in the formula. Thanks for updating the patches! They look good to me except that the formulas don't include the number of background processes requesting shared memory access, i.e., GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following formula in 9.3? ceil((max_connections + autovacuum_max_workers + number of background proceses + 5) / 16) Attached patch uses the above formula for 9.3. I'm thinking to push your patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. Comments? Regards, -- Fujii Masao
Вложения
Thanks for looking at this.
At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in
<CAHGQGwHEnT+-S+axWKQPBYSg6z852OfgS6gDXi0Ycpq5QW=iww@mail.gmail.com>
> >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
> >> under the Table 17-1.
> >
> > Oops! Thank you for pointing it out.
> >
> > The original description doesn't mention the magic-number ('5' in
> > the above formulas, which previously was '4') so I haven't added
> > anything about it.
> >
> > Process of which the number is limited by max_worker_processes is
> > called 'background process' (not 'backend worker') in the
> > documentation so I used the name to mention it in the additional
> > description.
> >
> > The difference in the body text for 9.2, 9.3 is only a literal
> > '4' to '5' in the formula.
>
> Thanks for updating the patches!
>
> They look good to me except that the formulas don't include the number of
> background processes requesting shared memory access, i.e.,
> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
> formula in 9.3?
>
> ceil((max_connections + autovacuum_max_workers + number of
> background proceses + 5) / 16)
>
> Attached patch uses the above formula for 9.3. I'm thinking to push your
> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
> Comments?
One concern is that users don't have any simple way to know how
many bg-workers a server runs in their current configuration.
The formula donsn't make sense without it.
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Mon, Feb 15, 2016 at 2:31 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> Thanks for looking at this.
>
> At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in
<CAHGQGwHEnT+-S+axWKQPBYSg6z852OfgS6gDXi0Ycpq5QW=iww@mail.gmail.com>
>> >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS
>> >> under the Table 17-1.
>> >
>> > Oops! Thank you for pointing it out.
>> >
>> > The original description doesn't mention the magic-number ('5' in
>> > the above formulas, which previously was '4') so I haven't added
>> > anything about it.
>> >
>> > Process of which the number is limited by max_worker_processes is
>> > called 'background process' (not 'backend worker') in the
>> > documentation so I used the name to mention it in the additional
>> > description.
>> >
>> > The difference in the body text for 9.2, 9.3 is only a literal
>> > '4' to '5' in the formula.
>>
>> Thanks for updating the patches!
>>
>> They look good to me except that the formulas don't include the number of
>> background processes requesting shared memory access, i.e.,
>> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following
>> formula in 9.3?
>>
>> ceil((max_connections + autovacuum_max_workers + number of
>> background proceses + 5) / 16)
>>
>> Attached patch uses the above formula for 9.3. I'm thinking to push your
>> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3.
>> Comments?
>
> One concern is that users don't have any simple way to know how
> many bg-workers a server runs in their current configuration.
Users need to read the document of the extensions they want to load,
to see the number of background worker processes which will be running.
> The formula donsn't make sense without it.
IMO, documenting "incorrect" formula can cause more troubles.
Regards,
--
Fujii Masao
On Fri, Feb 12, 2016 at 11:19 PM, Fujii Masao <masao.fujii@gmail.com> wrote: > On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI > <horiguchi.kyotaro@lab.ntt.co.jp> wrote: >> At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in <CAHGQGwHgBsat29_ZqK3aXg4a5Lsa0JUv579VkGWX3R_g0KOncw@mail.gmail.com> >>> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI >>> <horiguchi.kyotaro@lab.ntt.co.jp> wrote: >>> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS >>> > are incorrect in 9.2 and later. >>> > >>> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html >>> > >>> > But actually the number of semaphores PostgreSQL needs is >>> > calculated as following in 9.4 and later. >> ... >>> > So, the formula for SEMMNI should be >>> > >>> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) >>> > >>> > and SEMMNS should have the same fix. >>> > >>> > >>> > In 9.3 and 9.2, the documentation says the same thing but >> ... >>> > ceil((max_connections + autovacuum_max_workers + 5) / 16) >>> > >>> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. >>> >>> Good catch! >> >> Thanks. >> >>> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS >>> under the Table 17-1. >> >> Oops! Thank you for pointing it out. >> >> The original description doesn't mention the magic-number ('5' in >> the above formulas, which previously was '4') so I haven't added >> anything about it. >> >> Process of which the number is limited by max_worker_processes is >> called 'background process' (not 'backend worker') in the >> documentation so I used the name to mention it in the additional >> description. >> >> The difference in the body text for 9.2, 9.3 is only a literal >> '4' to '5' in the formula. > > Thanks for updating the patches! > > They look good to me except that the formulas don't include the number of > background processes requesting shared memory access, i.e., > GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following > formula in 9.3? > > ceil((max_connections + autovacuum_max_workers + number of > background proceses + 5) / 16) > > Attached patch uses the above formula for 9.3. I'm thinking to push your > patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. > Comments? Pushed. Thanks for the report and patches! Regards, -- Fujii Masao
Thank you for pushing this. At Tue, 16 Feb 2016 15:07:32 +0900, Fujii Masao <masao.fujii@gmail.com> wrote in <CAHGQGwGWhkDECnSB3GySt0PXigd8CqY7oj=sfUDz3ur4CmdZOw@mail.gmail.com> > > Attached patch uses the above formula for 9.3. I'm thinking to push your > > patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. > > Comments? > > Pushed. Thanks for the report and patches! regards, -- Kyotaro Horiguchi NTT Open Source Software Center