Re: Postgres Crash Issue

On Tue, Mar 15, 2022 at 07:20:57AM +0000, Sankar, Uma (Uma) wrote:
> Hi All,
> 
> We have a database crash issue last Friday and it's a patroni HA-based
> Postgres database running,  we have checked the pg logs and it shows
> error as shared memory corruptions. Can someone please check the
> attached logs and share your suggestions and feedback on the crash
> issues. Thanks in advance.

It wasn't crash.

Logs show:

#v+
2022-03-11 08:58:42.956 UTC [27799] LOG:  received immediate shutdown request
2022-03-11 08:58:42.956 UTC [17115] WARNING:  terminating connection because of crash of another server process
2022-03-11 08:58:42.956 UTC [17115] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted shared memory.
 
2022-03-11 08:58:42.956 UTC [17115] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2022-03-11 08:58:42.957 UTC [14618] WARNING:  terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [14618] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted shared memory.
 
2022-03-11 08:58:42.957 UTC [14618] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2022-03-11 08:58:42.957 UTC [16396] WARNING:  terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [16396] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted shared memory.
 
2022-03-11 08:58:42.957 UTC [16396] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2022-03-11 08:58:42.957 UTC [12873] WARNING:  terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [12873] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted shared memory.
 
2022-03-11 08:58:42.957 UTC [12873] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
2022-03-11 08:58:42.958 UTC [16564] WARNING:  terminating connection because of crash of another server process
2022-03-11 08:58:42.958 UTC [16564] DETAIL:  The postmaster has commanded this server process to roll back the current
transactionand exit, because another server process exited abnormally and possibly corrupted shared memory.
 
2022-03-11 08:58:42.958 UTC [16564] HINT:  In a moment you should be able to reconnect to the database and repeat your
command.
#v-

and

#v+
2022-03-11 08:58:45.969 UTC [17354] LOG:  consistent recovery state reached at A/E1836620
2022-03-11 08:58:45.969 UTC [17347] LOG:  database system is ready to accept read only connections
2022-03-11 08:59:01.234 UTC [17347] LOG:  received fast shutdown request
2022-03-11 08:59:01.234 UTC [17347] LOG:  aborting any active transactions
2022-03-11 08:59:01.234 UTC [17412] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.234 UTC [17415] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.234 UTC [17393] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.235 UTC [17399] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.235 UTC [17408] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.236 UTC [17394] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.236 UTC [17389] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.237 UTC [17400] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.237 UTC [17378] FATAL:  terminating connection due to administrator command
2022-03-11 08:59:01.238 UTC [17362] LOG:  shutting down
2022-03-11 08:59:01.254 UTC [17347] LOG:  database system is shut down
2022-03-11 08:59:01.419 UTC [17444] LOG:  database system was shut down in recovery at 2022-03-11 08:59:01 UTC
2022-03-11 08:59:01.419 UTC [17444] LOG:  entering standby mode
2022-03-11 08:59:01.420 UTC [17444] LOG:  redo starts at A/E17CB690
#v-

Something/someone stopped the pg, twice, ~ 20 seconds apart.
What/how/why - can't tell, but "received ... shutdown request" is not
crash. It is someone/something doing "pg_ctl stop"

depesz