Thank you Tom for detailed instructions!
What I understood is that some specific query lead to database block corruption which causes segfault.
I analyzed action log before segfault
1. database worked fine on pg13.0 for a few months
2. issue caused right after pg13.1 binaries upgrade
3. (this actually was wrong action) rollback pg13.0 binaries and tried to start database on it and now got infinite segfault on startup
LOG: database system was interrupted while in recovery at 2021-01-22 14:27:49 UTC
HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 49D/364DD528
LOG: startup process (PID 26) was terminated by signal 11: Segmentation fault
LOG: aborting startup due to startup process failure
LOG: database system is shut down
Most disappointed fact that slave node also got corrupted wal block and also unable to start.
So I have a chance to recover database with initdb+pgdump only.
Anyway I will try to compile pg13.1 binaries with --enable-debug and enable all queries logging. Hope this will help with the investigation.
Thanks for your support!
PG Bug reporting form <noreply@postgresql.org> writes:
> Process crash inside docker containter 2-3 times per hour without any
> additional information
> ./postgresql-Thu-00.log:2021-01-21 00:11:34 UTC [1]: user=,db=,app=,client=
> LOG: server process (PID 20071) was terminated by signal 11: Segmentation
> fault
Hm, please see if you can get a stack trace:
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
Also try to figure out what query(s) are causing the crash.
(It's unlikely that the postmaster log doesn't provide more
information than you've shared here.)
> I can share postgresql.conf, process crash core dumps for analysis
Core dumps are unlikely to help anyone else; they are too
machine-specific. Not to mention that they might contain
sensitive data. You'll need to examine them yourself.
regards, tom lane