Обсуждение: [ADMIN] WARNING: terminating connection because of crash of another serverprocess

Поиск
Список
Период
Сортировка

[ADMIN] WARNING: terminating connection because of crash of another serverprocess

От
"Saeed Ahmed (DB)"
Дата:

Dear Team,

 

 

We are having POSGRESQL version 9.6  installed on OS version Linux REDHAT 7.2. While running our Decoder (for larger load) OR Audit execution, we are seeing connection errors as service to DB goes down when application tries to connect to postgres.

 

Log file is attached.

 

 

 

 

2017-09-14 00:35:16 ICT [5769]: [1-1] user=,db=,app=,client= LOG:  checkpoint starting: time

2017-09-14 00:35:16 ICT [5769]: [2-1] user=,db=,app=,client= LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.486 s, sync=0.000 s, total=0.491 s; sync files=3, longest=0.000 s, average=0.000 s; distance=23 kB, estimate=23 kB

2017-09-14 00:39:01 ICT [5803]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=[local] LOG:  connection received: host=[local]

2017-09-14 00:39:01 ICT [5803]: [2-1] user=postgres,db=postgres,app=[unknown],client=[local] LOG:  connection authorized: user=postgres database=postgres

2017-09-14 00:39:01 ICT [5803]: [3-1] user=postgres,db=postgres,app=psql,client=[local] LOG:  disconnection: session time: 0:00:00.008 user=postgres database=postgres host=[local]

2017-09-14 00:39:16 ICT [5812]: [1-1] user=,db=,app=,client= FATAL:  semctl(1693253819, 10, SETVAL, 0) failed: Invalid argument

2017-09-14 00:39:16 ICT [1617]: [13584-1] user=,db=,app=,client= LOG:  server process (PID 5812) exited with exit code 1

2017-09-14 00:39:16 ICT [1617]: [13585-1] user=,db=,app=,client= LOG:  terminating any other active server processes

2017-09-14 00:39:16 ICT [5782]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5782]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5782]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5783]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5783]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5783]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5779]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5779]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5779]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5780]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5780]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5780]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5781]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5781]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5781]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5777]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5777]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5777]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5778]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5778]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 

Regards,

 

Saeed Ahmed

 

Вложения

[ADMIN] Re: WARNING: terminating connection because of crash of anotherserver process

От
Scott Whitney
Дата:

It looks like maybe your semaphores are getting deleted out from under PG as a guess.


From: pgsql-admin-owner@postgresql.org <pgsql-admin-owner@postgresql.org> on behalf of Saeed Ahmed (DB) <saeed.ahmed1@telenor.com.pk>
Sent: Monday, September 18, 2017 7:32 AM
To: pgsql-admin@postgresql.org
Subject: [ADMIN] WARNING: terminating connection because of crash of another server process
 

Dear Team,

 

 

We are having POSGRESQL version 9.6  installed on OS version Linux REDHAT 7.2. While running our Decoder (for larger load) OR Audit execution, we are seeing connection errors as service to DB goes down when application tries to connect to postgres.

 

Log file is attached.

 

 

 

 

2017-09-14 00:35:16 ICT [5769]: [1-1] user=,db=,app=,client= LOG:  checkpoint starting: time

2017-09-14 00:35:16 ICT [5769]: [2-1] user=,db=,app=,client= LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.486 s, sync=0.000 s, total=0.491 s; sync files=3, longest=0.000 s, average=0.000 s; distance=23 kB, estimate=23 kB

2017-09-14 00:39:01 ICT [5803]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=[local] LOG:  connection received: host=[local]

2017-09-14 00:39:01 ICT [5803]: [2-1] user=postgres,db=postgres,app=[unknown],client=[local] LOG:  connection authorized: user=postgres database=postgres

2017-09-14 00:39:01 ICT [5803]: [3-1] user=postgres,db=postgres,app=psql,client=[local] LOG:  disconnection: session time: 0:00:00.008 user=postgres database=postgres host=[local]

2017-09-14 00:39:16 ICT [5812]: [1-1] user=,db=,app=,client= FATAL:  semctl(1693253819, 10, SETVAL, 0) failed: Invalid argument

2017-09-14 00:39:16 ICT [1617]: [13584-1] user=,db=,app=,client= LOG:  server process (PID 5812) exited with exit code 1

2017-09-14 00:39:16 ICT [1617]: [13585-1] user=,db=,app=,client= LOG:  terminating any other active server processes

2017-09-14 00:39:16 ICT [5782]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5782]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5782]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5783]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5783]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5783]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5779]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5779]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5779]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5780]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5780]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5780]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5781]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5781]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5781]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5777]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5777]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2017-09-14 00:39:16 ICT [5777]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2017-09-14 00:39:16 ICT [5778]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING:  terminating connection because of crash of another server process

2017-09-14 00:39:16 ICT [5778]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 

Regards,

 

Saeed Ahmed

 



Journyx, Inc.
7600 Burnet Road #300
Austin, TX 78757
www.journyx.com

p 512.834.8888 
f 512-834-8858 

Do you receive our promotional emails? Click here or visit http://journyx.com/communication-preferences to unsubscribe.

Re: [ADMIN] Re: WARNING: terminating connection because of crash of another server process

От
Tom Lane
Дата:
Scott Whitney <scott@journyx.com> writes:
> It looks like maybe your semaphores are getting deleted out from under PG as a guess.

Yeah.  See

https://www.postgresql.org/docs/devel/static/kernel-resources.html#systemd-removeipc

(Probably we should back-patch that section into older versions of the
docs, as it's not at all PG-version-specific.)
        regards, tom lane


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [ADMIN] Re: WARNING: terminating connection because of crash ofanother server process

От
Peter Eisentraut
Дата:
On 9/19/17 11:18, Tom Lane wrote:
> Scott Whitney <scott@journyx.com> writes:
>> It looks like maybe your semaphores are getting deleted out from under PG as a guess.
> 
> Yeah.  See
> 
> https://www.postgresql.org/docs/devel/static/kernel-resources.html#systemd-removeipc
> 
> (Probably we should back-patch that section into older versions of the
> docs, as it's not at all PG-version-specific.)

done

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services