Обсуждение: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size

I remember an old issue I had with postgres and shared memory being set too low by the distro I was using.

What's the value of "/proc/sys/kernel/shmmax"? Try setting to 18446744073709551615

Regards,

Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.

Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe

----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>
To: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 14:44:08
Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size

We recently upgraded a database that a number of developers use from
Postgres 9.4 to 10.  Our memory parameters are unchanged.

We are seeing a lot of errors that look like this in the postgres log:

< 2017-11-14 13:13:59 EST [144172] : [1-1]
user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL:  no PostgreSQL
user name specified in startup packet
< 2017-11-14 13:52:22 EST [6273] : [1-1]
user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR:  out of
shared memory
< 2017-11-14 14:17:53 EST [6200] : [3-1]
user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
invalid memory alloc request size 34359738368

Anecdotally it seems that these errors are happening when
powa_take_snapshot() (Postgres workload analyzer) is running as well but I
can't confirm that.

Does anyone know how to debug these types of errors?  And once the db is in
this state, how do we resolve sessions reporting "Error: invalid memory
alloc request size" - killing the sessions and having them reconnect does
not fix the issue.

Thanks,
Arjun


-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Try with:

echo "18446744073709551615" > /proc/sys/kernel/shmax

Then restart postgres

Regards,

Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.

Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe

----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>
To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com>
Cc: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 15:00:03
Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size

/proc/sys/kernel/shmax = 68719476736


On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
aaguayo@opensysperu.com> wrote:

> I remember an old issue I had with postgres and shared memory being set
> too low by the distro I was using.
>
> What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> 18446744073709551615
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com>
> To: "pgsql-admin" <pgsql-admin@postgresql.org>
> Sent: Tuesday, 14 November, 2017 14:44:08
> Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> request size
>
> We recently upgraded a database that a number of developers use from
> Postgres 9.4 to 10.  Our memory parameters are unchanged.
>
> We are seeing a lot of errors that look like this in the postgres log:
>
> < 2017-11-14 13:13:59 EST [144172] : [1-1]
> user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL:  no
> PostgreSQL
> user name specified in startup packet
> < 2017-11-14 13:52:22 EST [6273] : [1-1]
> user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR:  out of
> shared memory
> < 2017-11-14 14:17:53 EST [6200] : [3-1]
> user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
> invalid memory alloc request size 34359738368
>
> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but I
> can't confirm that.
>
> Does anyone know how to debug these types of errors?  And once the db is in
> this state, how do we resolve sessions reporting "Error: invalid memory
> alloc request size" - killing the sessions and having them reconnect does
> not fix the issue.
>
> Thanks,
> Arjun
>


-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Arjun Ranade <ranade@nodalexchange.com> writes:
> We will try that.  I am wondering why we need to worry about this in pg10
> whereas we had no issues with shared memory in 9.4

What it looks like to me is something is miscomputing a memory request
size and asking for some silly amount of memory.  I think you're dealing
with a software bug, not inadequate configuration.

> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but
> I can't confirm that.

The first theory that comes to mind is that powa_take_snapshot() requires
some updates for PG10.  I do not know what that function is or where you
got it from, but if there's any C code involved, you need to discuss this
with its developer(s).
        regards, tom lane


-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

I don't think this is the issue (Tom's theory sounds more likely). Also 
changing the IPC shared memory limit is not going to make any difference 
for these later versions of Postgres (>= 9.3) as it uses POSIX shared 
memory (I believe the whole point of that change was to avoid the need 
to tweak these IPC parameters)!

regards

Mark


On 15/11/17 09:07, Arjun Ranade wrote:
> We will try that.  I am wondering why we need to worry about this in 
> pg10 whereas we had no issues with shared memory in 9.4
>
> Thanks,
> Arjun
>
> On Tue, Nov 14, 2017 at 3:05 PM, Alvaro Aguayo Garcia-Rada 
> <aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote:
>
>     Try with:
>
>     echo "18446744073709551615" > /proc/sys/kernel/shmax
>
>     Then restart postgres
>
>     Regards,
>
>     Alvaro Aguayo
>     Operations Manager
>     Open Comb Systems E.I.R.L.
>
>     Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile:
>     (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248
>     <tel:%28%2B51%29%20954183248>
>     Web: www.ocs.pe <http://www.ocs.pe>
>
>     ----- Original Message -----
>     From: "Arjun Ranade" <ranade@nodalexchange.com
>     <mailto:ranade@nodalexchange.com>>
>     To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com
>     <mailto:aaguayo@opensysperu.com>>
>     Cc: "pgsql-admin" <pgsql-admin@postgresql.org
>     <mailto:pgsql-admin@postgresql.org>>
>     Sent: Tuesday, 14 November, 2017 15:00:03
>     Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory
>     alloc request size
>
>     /proc/sys/kernel/shmax = 68719476736
>
>
>     On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
>     aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote:
>
>     > I remember an old issue I had with postgres and shared memory
>     being set
>     > too low by the distro I was using.
>     >
>     > What's the value of "/proc/sys/kernel/shmmax"? Try setting to
>     > 18446744073709551615
>     >
>     > Regards,
>     >
>     > Alvaro Aguayo
>     > Operations Manager
>     > Open Comb Systems E.I.R.L.
>     >
>     > Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile:
>     (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248
>     <tel:%28%2B51%29%20954183248>
>     > Web: www.ocs.pe <http://www.ocs.pe>
>     >
>     > ----- Original Message -----
>     > From: "Arjun Ranade" <ranade@nodalexchange.com
>     <mailto:ranade@nodalexchange.com>>
>     > To: "pgsql-admin" <pgsql-admin@postgresql.org
>     <mailto:pgsql-admin@postgresql.org>>
>     > Sent: Tuesday, 14 November, 2017 14:44:08
>     > Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
>     > request size
>     >
>     > We recently upgraded a database that a number of developers use from
>     > Postgres 9.4 to 10.  Our memory parameters are unchanged.
>     >
>     > We are seeing a lot of errors that look like this in the
>     postgres log:
>     >
>     > < 2017-11-14 13:13:59 EST [144172] : [1-1]
>     > user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL:  no
>     > PostgreSQL
>     > user name specified in startup packet
>     > < 2017-11-14 13:52:22 EST [6273] : [1-1]
>     > user=deployment_app,db=qa5,remote=192.168.104.83(45384) >
>     ERROR:  out of
>     > shared memory
>     > < 2017-11-14 14:17:53 EST [6200] : [3-1]
>     > user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) >
>     ERROR:
>     > invalid memory alloc request size 34359738368
>     >
>     > Anecdotally it seems that these errors are happening when
>     > powa_take_snapshot() (Postgres workload analyzer) is running as
>     well but I
>     > can't confirm that.
>     >
>     > Does anyone know how to debug these types of errors?  And once
>     the db is in
>     > this state, how do we resolve sessions reporting "Error: invalid
>     memory
>     > alloc request size" - killing the sessions and having them
>     reconnect does
>     > not fix the issue.
>     >
>     > Thanks,
>     > Arjun
>     >
>
>



-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Sorry: I've blithely typed IPC when I should have said SYS V IPC...


On 15/11/17 12:24, Mark Kirkwood wrote:
> I don't think this is the issue (Tom's theory sounds more likely). 
> Also changing the IPC shared memory limit is not going to make any 
> difference for these later versions of Postgres (>= 9.3) as it uses 
> POSIX shared memory (I believe the whole point of that change was to 
> avoid the need to tweak these IPC parameters)!



-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin