Обсуждение: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size
We recently upgraded a database that a number of developers use from Postgres 9.4 to 10. Our memory parameters are unchanged.
We are seeing a lot of errors that look like this in the postgres log:< 2017-11-14 13:13:59 EST [144172] : [1-1] user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no PostgreSQL user name specified in startup packet
< 2017-11-14 13:52:22 EST [6273] : [1-1] user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of shared memory
< 2017-11-14 14:17:53 EST [6200] : [3-1] user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR: invalid memory alloc request size 34359738368
Re: [ADMIN] PG10 - Out of shared memory / invalid memory allocrequest size
От
Alvaro Aguayo Garcia-Rada
Дата:
I remember an old issue I had with postgres and shared memory being set too low by the distro I was using. What's the value of "/proc/sys/kernel/shmmax"? Try setting to 18446744073709551615 Regards, Alvaro Aguayo Operations Manager Open Comb Systems E.I.R.L. Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248 Web: www.ocs.pe ----- Original Message ----- From: "Arjun Ranade" <ranade@nodalexchange.com> To: "pgsql-admin" <pgsql-admin@postgresql.org> Sent: Tuesday, 14 November, 2017 14:44:08 Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size We recently upgraded a database that a number of developers use from Postgres 9.4 to 10. Our memory parameters are unchanged. We are seeing a lot of errors that look like this in the postgres log: < 2017-11-14 13:13:59 EST [144172] : [1-1] user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no PostgreSQL user name specified in startup packet < 2017-11-14 13:52:22 EST [6273] : [1-1] user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of shared memory < 2017-11-14 14:17:53 EST [6200] : [3-1] user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR: invalid memory alloc request size 34359738368 Anecdotally it seems that these errors are happening when powa_take_snapshot() (Postgres workload analyzer) is running as well but I can't confirm that. Does anyone know how to debug these types of errors? And once the db is in this state, how do we resolve sessions reporting "Error: invalid memory alloc request size" - killing the sessions and having them reconnect does not fix the issue. Thanks, Arjun -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
/proc/sys/kernel/shmax = 68719476736
On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com> wrote:
I remember an old issue I had with postgres and shared memory being set too low by the distro I was using.
What's the value of "/proc/sys/kernel/shmmax"? Try setting to 18446744073709551615
Regards,
Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.
Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe
----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>
To: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 14:44:08
Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size
We recently upgraded a database that a number of developers use from
Postgres 9.4 to 10. Our memory parameters are unchanged.
We are seeing a lot of errors that look like this in the postgres log:
< 2017-11-14 13:13:59 EST [144172] : [1-1]
user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no PostgreSQL
user name specified in startup packet
< 2017-11-14 13:52:22 EST [6273] : [1-1]
user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
shared memory
< 2017-11-14 14:17:53 EST [6200] : [3-1]
user=service_tracker_app,db=qa16,remote=192.168.104.22( 46996) > ERROR:
invalid memory alloc request size 34359738368
Anecdotally it seems that these errors are happening when
powa_take_snapshot() (Postgres workload analyzer) is running as well but I
can't confirm that.
Does anyone know how to debug these types of errors? And once the db is in
this state, how do we resolve sessions reporting "Error: invalid memory
alloc request size" - killing the sessions and having them reconnect does
not fix the issue.
Thanks,
Arjun
Re: [ADMIN] PG10 - Out of shared memory / invalid memory allocrequest size
От
Alvaro Aguayo Garcia-Rada
Дата:
Try with: echo "18446744073709551615" > /proc/sys/kernel/shmax Then restart postgres Regards, Alvaro Aguayo Operations Manager Open Comb Systems E.I.R.L. Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248 Web: www.ocs.pe ----- Original Message ----- From: "Arjun Ranade" <ranade@nodalexchange.com> To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com> Cc: "pgsql-admin" <pgsql-admin@postgresql.org> Sent: Tuesday, 14 November, 2017 15:00:03 Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size /proc/sys/kernel/shmax = 68719476736 On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada < aaguayo@opensysperu.com> wrote: > I remember an old issue I had with postgres and shared memory being set > too low by the distro I was using. > > What's the value of "/proc/sys/kernel/shmmax"? Try setting to > 18446744073709551615 > > Regards, > > Alvaro Aguayo > Operations Manager > Open Comb Systems E.I.R.L. > > Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248 > Web: www.ocs.pe > > ----- Original Message ----- > From: "Arjun Ranade" <ranade@nodalexchange.com> > To: "pgsql-admin" <pgsql-admin@postgresql.org> > Sent: Tuesday, 14 November, 2017 14:44:08 > Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc > request size > > We recently upgraded a database that a number of developers use from > Postgres 9.4 to 10. Our memory parameters are unchanged. > > We are seeing a lot of errors that look like this in the postgres log: > > < 2017-11-14 13:13:59 EST [144172] : [1-1] > user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no > PostgreSQL > user name specified in startup packet > < 2017-11-14 13:52:22 EST [6273] : [1-1] > user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of > shared memory > < 2017-11-14 14:17:53 EST [6200] : [3-1] > user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR: > invalid memory alloc request size 34359738368 > > Anecdotally it seems that these errors are happening when > powa_take_snapshot() (Postgres workload analyzer) is running as well but I > can't confirm that. > > Does anyone know how to debug these types of errors? And once the db is in > this state, how do we resolve sessions reporting "Error: invalid memory > alloc request size" - killing the sessions and having them reconnect does > not fix the issue. > > Thanks, > Arjun > -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
We will try that. I am wondering why we need to worry about this in pg10 whereas we had no issues with shared memory in 9.4
Thanks,On Tue, Nov 14, 2017 at 3:05 PM, Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com> wrote:
Try with:
echo "18446744073709551615" > /proc/sys/kernel/shmax
Then restart postgres
Regards,
Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.
Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe
----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com>
Cc: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 15:00:03
Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size
/proc/sys/kernel/shmax = 68719476736
On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
aaguayo@opensysperu.com> wrote:
> I remember an old issue I had with postgres and shared memory being set
> too low by the distro I was using.
>
> What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> 18446744073709551615
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com>
> To: "pgsql-admin" <pgsql-admin@postgresql.org>
> Sent: Tuesday, 14 November, 2017 14:44:08
> Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> request size
>
> We recently upgraded a database that a number of developers use from
> Postgres 9.4 to 10. Our memory parameters are unchanged.
>
> We are seeing a lot of errors that look like this in the postgres log:
>
> < 2017-11-14 13:13:59 EST [144172] : [1-1]
> user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no
> PostgreSQL
> user name specified in startup packet
> < 2017-11-14 13:52:22 EST [6273] : [1-1]
> user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
> shared memory
> < 2017-11-14 14:17:53 EST [6200] : [3-1]
> user=service_tracker_app,db=qa16,remote=192.168.104.22( 46996) > ERROR:
> invalid memory alloc request size 34359738368
>
> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but I
> can't confirm that.
>
> Does anyone know how to debug these types of errors? And once the db is in
> this state, how do we resolve sessions reporting "Error: invalid memory
> alloc request size" - killing the sessions and having them reconnect does
> not fix the issue.
>
> Thanks,
> Arjun
>
Arjun Ranade <ranade@nodalexchange.com> writes: > We will try that. I am wondering why we need to worry about this in pg10 > whereas we had no issues with shared memory in 9.4 What it looks like to me is something is miscomputing a memory request size and asking for some silly amount of memory. I think you're dealing with a software bug, not inadequate configuration. > Anecdotally it seems that these errors are happening when > powa_take_snapshot() (Postgres workload analyzer) is running as well but > I can't confirm that. The first theory that comes to mind is that powa_take_snapshot() requires some updates for PG10. I do not know what that function is or where you got it from, but if there's any C code involved, you need to discuss this with its developer(s). regards, tom lane -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
I don't think this is the issue (Tom's theory sounds more likely). Also changing the IPC shared memory limit is not going to make any difference for these later versions of Postgres (>= 9.3) as it uses POSIX shared memory (I believe the whole point of that change was to avoid the need to tweak these IPC parameters)! regards Mark On 15/11/17 09:07, Arjun Ranade wrote: > We will try that. I am wondering why we need to worry about this in > pg10 whereas we had no issues with shared memory in 9.4 > > Thanks, > Arjun > > On Tue, Nov 14, 2017 at 3:05 PM, Alvaro Aguayo Garcia-Rada > <aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote: > > Try with: > > echo "18446744073709551615" > /proc/sys/kernel/shmax > > Then restart postgres > > Regards, > > Alvaro Aguayo > Operations Manager > Open Comb Systems E.I.R.L. > > Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile: > (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248 > <tel:%28%2B51%29%20954183248> > Web: www.ocs.pe <http://www.ocs.pe> > > ----- Original Message ----- > From: "Arjun Ranade" <ranade@nodalexchange.com > <mailto:ranade@nodalexchange.com>> > To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com > <mailto:aaguayo@opensysperu.com>> > Cc: "pgsql-admin" <pgsql-admin@postgresql.org > <mailto:pgsql-admin@postgresql.org>> > Sent: Tuesday, 14 November, 2017 15:00:03 > Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory > alloc request size > > /proc/sys/kernel/shmax = 68719476736 > > > On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada < > aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote: > > > I remember an old issue I had with postgres and shared memory > being set > > too low by the distro I was using. > > > > What's the value of "/proc/sys/kernel/shmmax"? Try setting to > > 18446744073709551615 > > > > Regards, > > > > Alvaro Aguayo > > Operations Manager > > Open Comb Systems E.I.R.L. > > > > Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile: > (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248 > <tel:%28%2B51%29%20954183248> > > Web: www.ocs.pe <http://www.ocs.pe> > > > > ----- Original Message ----- > > From: "Arjun Ranade" <ranade@nodalexchange.com > <mailto:ranade@nodalexchange.com>> > > To: "pgsql-admin" <pgsql-admin@postgresql.org > <mailto:pgsql-admin@postgresql.org>> > > Sent: Tuesday, 14 November, 2017 14:44:08 > > Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc > > request size > > > > We recently upgraded a database that a number of developers use from > > Postgres 9.4 to 10. Our memory parameters are unchanged. > > > > We are seeing a lot of errors that look like this in the > postgres log: > > > > < 2017-11-14 13:13:59 EST [144172] : [1-1] > > user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no > > PostgreSQL > > user name specified in startup packet > > < 2017-11-14 13:52:22 EST [6273] : [1-1] > > user=deployment_app,db=qa5,remote=192.168.104.83(45384) > > ERROR: out of > > shared memory > > < 2017-11-14 14:17:53 EST [6200] : [3-1] > > user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > > ERROR: > > invalid memory alloc request size 34359738368 > > > > Anecdotally it seems that these errors are happening when > > powa_take_snapshot() (Postgres workload analyzer) is running as > well but I > > can't confirm that. > > > > Does anyone know how to debug these types of errors? And once > the db is in > > this state, how do we resolve sessions reporting "Error: invalid > memory > > alloc request size" - killing the sessions and having them > reconnect does > > not fix the issue. > > > > Thanks, > > Arjun > > > > -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Sorry: I've blithely typed IPC when I should have said SYS V IPC... On 15/11/17 12:24, Mark Kirkwood wrote: > I don't think this is the issue (Tom's theory sounds more likely). > Also changing the IPC shared memory limit is not going to make any > difference for these later versions of Postgres (>= 9.3) as it uses > POSIX shared memory (I believe the whole point of that change was to > avoid the need to tweak these IPC parameters)! -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin