Обсуждение: [ADMIN] WARNING: terminating connection because of crash of another serverprocess
[ADMIN] WARNING: terminating connection because of crash of another serverprocess
Dear Team,
We are having POSGRESQL version 9.6 installed on OS version Linux REDHAT 7.2. While running our Decoder (for larger load) OR Audit execution, we are seeing connection errors as service to DB goes down when application tries to connect to postgres.
Log file is attached.
2017-09-14 00:35:16 ICT [5769]: [1-1] user=,db=,app=,client= LOG: checkpoint starting: time
2017-09-14 00:35:16 ICT [5769]: [2-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.486 s, sync=0.000 s, total=0.491 s; sync files=3, longest=0.000 s, average=0.000 s; distance=23 kB, estimate=23 kB
2017-09-14 00:39:01 ICT [5803]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=[local] LOG: connection received: host=[local]
2017-09-14 00:39:01 ICT [5803]: [2-1] user=postgres,db=postgres,app=[unknown],client=[local] LOG: connection authorized: user=postgres database=postgres
2017-09-14 00:39:01 ICT [5803]: [3-1] user=postgres,db=postgres,app=psql,client=[local] LOG: disconnection: session time: 0:00:00.008 user=postgres database=postgres host=[local]
2017-09-14 00:39:16 ICT [5812]: [1-1] user=,db=,app=,client= FATAL: semctl(1693253819, 10, SETVAL, 0) failed: Invalid argument
2017-09-14 00:39:16 ICT [1617]: [13584-1] user=,db=,app=,client= LOG: server process (PID 5812) exited with exit code 1
2017-09-14 00:39:16 ICT [1617]: [13585-1] user=,db=,app=,client= LOG: terminating any other active server processes
2017-09-14 00:39:16 ICT [5782]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5782]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5782]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5783]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5783]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5783]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5779]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5779]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5779]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5780]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5780]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5780]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5781]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5781]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5781]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5777]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5777]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5777]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5778]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5778]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Regards,
Saeed Ahmed
Вложения
[ADMIN] Re: WARNING: terminating connection because of crash of anotherserver process
It looks like maybe your semaphores are getting deleted out from under PG as a guess.
Sent: Monday, September 18, 2017 7:32 AM
To: pgsql-admin@postgresql.org
Subject: [ADMIN] WARNING: terminating connection because of crash of another server process
Dear Team,
We are having POSGRESQL version 9.6 installed on OS version Linux REDHAT 7.2. While running our Decoder (for larger load) OR Audit execution, we are seeing connection errors as service to DB goes down when application tries to connect to postgres.
Log file is attached.
2017-09-14 00:35:16 ICT [5769]: [1-1] user=,db=,app=,client= LOG: checkpoint starting: time
2017-09-14 00:35:16 ICT [5769]: [2-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.486 s, sync=0.000 s, total=0.491 s; sync files=3, longest=0.000 s, average=0.000 s; distance=23 kB, estimate=23 kB
2017-09-14 00:39:01 ICT [5803]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=[local] LOG: connection received: host=[local]
2017-09-14 00:39:01 ICT [5803]: [2-1] user=postgres,db=postgres,app=[unknown],client=[local] LOG: connection authorized: user=postgres database=postgres
2017-09-14 00:39:01 ICT [5803]: [3-1] user=postgres,db=postgres,app=psql,client=[local] LOG: disconnection: session time: 0:00:00.008 user=postgres database=postgres host=[local]
2017-09-14 00:39:16 ICT [5812]: [1-1] user=,db=,app=,client= FATAL: semctl(1693253819, 10, SETVAL, 0) failed: Invalid argument
2017-09-14 00:39:16 ICT [1617]: [13584-1] user=,db=,app=,client= LOG: server process (PID 5812) exited with exit code 1
2017-09-14 00:39:16 ICT [1617]: [13585-1] user=,db=,app=,client= LOG: terminating any other active server processes
2017-09-14 00:39:16 ICT [5782]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5782]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5782]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5783]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5783]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5783]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5779]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5779]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5779]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5780]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5780]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5780]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5781]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5781]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5781]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5777]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5777]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-09-14 00:39:16 ICT [5777]: [5-1] user=raref,db=radb1,app=[unknown],client=10.68.128.1 HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-09-14 00:39:16 ICT [5778]: [3-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 WARNING: terminating connection because of crash of another server process
2017-09-14 00:39:16 ICT [5778]: [4-1] user=raref,db=radb1,app=[unknown],client=10.68.128.7 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Regards,
Saeed Ahmed
Austin, TX 78757
www.journyx.com
Scott Whitney <scott@journyx.com> writes: > It looks like maybe your semaphores are getting deleted out from under PG as a guess. Yeah. See https://www.postgresql.org/docs/devel/static/kernel-resources.html#systemd-removeipc (Probably we should back-patch that section into older versions of the docs, as it's not at all PG-version-specific.) regards, tom lane -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Re: WARNING: terminating connection because of crash ofanother server process
On 9/19/17 11:18, Tom Lane wrote: > Scott Whitney <scott@journyx.com> writes: >> It looks like maybe your semaphores are getting deleted out from under PG as a guess. > > Yeah. See > > https://www.postgresql.org/docs/devel/static/kernel-resources.html#systemd-removeipc > > (Probably we should back-patch that section into older versions of the > docs, as it's not at all PG-version-specific.) done -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services