Обсуждение: How to debug "background worker "parallel worker" (PID 32152) wasterminated by signal 11: Segmentation fault"
Hi,
Can you guide me how to debug postgresql crash?
I have a development VM (kvm) running Centos 8.1(up to date with all patches, kernel 4.18.0-147.8.1.el8_1.x86_64, postgresql12-server-12.2-2PGDG.rhel8.x86_64 ) with a very large partitioned table (few hundred GBs) . Creating an index on partition tables crashed the postgresql instance. I have in the log:
2020-04-26 17:35:50.065 EEST [8385] LOG: background worker "parallel worker" (PID 32152) was terminated by signal 11: Segmentation fault
2020-04-26 17:35:50.065 EEST [8385] DETAIL: Failed process was running: CREATE INDEX IDX_instrid
ON ach.achiteminfo USING btree
(instrid);
2020-04-26 17:35:50.065 EEST [8385] LOG: terminating any other active server processes
2020-04-26 17:35:50.065 EEST [32134] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32134] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32134] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32146] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32146] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32146] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32126] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32126] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32126] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32130] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32130] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32130] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32129] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32129] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32129] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32132] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32132] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32132] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32142] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32142] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32142] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32143] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32143] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32143] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32140] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32140] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32140] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.069 EEST [32141] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.069 EEST [32141] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.069 EEST [32141] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.307 EEST [8385] LOG: all server processes terminated; reinitializing
2020-04-26 17:35:52.355 EEST [32164] LOG: database system was interrupted; last known up at 2020-04-26 17:30:29 EEST
2020-04-26 17:35:52.703 EEST [32164] LOG: database system was not properly shut down; automatic recovery in progress
2020-04-26 17:35:52.705 EEST [32164] LOG: redo starts at 6FD/A53E0110
2020-04-26 17:35:53.885 EEST [32165] FATAL: the database system is in recovery mode
2020-04-26 17:35:53.894 EEST [32166] FATAL: the database system is in recovery mode
2020-04-26 17:35:55.285 EEST [32167] FATAL: the database system is in recovery mode
2020-04-26 17:35:56.346 EEST [32168] FATAL: the database system is in recovery mode
2020-04-26 17:35:57.416 EEST [32169] FATAL: the database system is in recovery mode
2020-04-26 17:35:58.270 EEST [32164] LOG: redo done at 6FD/ED969B98
2020-04-26 17:35:58.466 EEST [32170] FATAL: the database system is in recovery mode
2020-04-26 17:35:59.555 EEST [32171] FATAL: the database system is in recovery mode
2020-04-26 17:36:00.606 EEST [32172] FATAL: the database system is in recovery mode
2020-04-26 17:36:02.746 EEST [32174] FATAL: the database system is in recovery mode
2020-04-26 17:36:03.409 EEST [8385] LOG: database system is ready to accept connections
2020-04-26 17:36:03.805 EEST [32175] FATAL: the database system is in recovery mode
2020-04-26 17:35:50.065 EEST [8385] DETAIL: Failed process was running: CREATE INDEX IDX_instrid
ON ach.achiteminfo USING btree
(instrid);
2020-04-26 17:35:50.065 EEST [8385] LOG: terminating any other active server processes
2020-04-26 17:35:50.065 EEST [32134] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32134] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32134] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32146] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32146] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32146] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.065 EEST [32126] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.065 EEST [32126] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.065 EEST [32126] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32130] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32130] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32130] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.066 EEST [32129] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.066 EEST [32129] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.066 EEST [32129] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32132] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32132] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32132] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.067 EEST [32142] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.067 EEST [32142] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.067 EEST [32142] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32143] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32143] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32143] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.068 EEST [32140] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.068 EEST [32140] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.068 EEST [32140] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.069 EEST [32141] WARNING: terminating connection because of crash of another server process
2020-04-26 17:35:50.069 EEST [32141] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-26 17:35:50.069 EEST [32141] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-04-26 17:35:50.307 EEST [8385] LOG: all server processes terminated; reinitializing
2020-04-26 17:35:52.355 EEST [32164] LOG: database system was interrupted; last known up at 2020-04-26 17:30:29 EEST
2020-04-26 17:35:52.703 EEST [32164] LOG: database system was not properly shut down; automatic recovery in progress
2020-04-26 17:35:52.705 EEST [32164] LOG: redo starts at 6FD/A53E0110
2020-04-26 17:35:53.885 EEST [32165] FATAL: the database system is in recovery mode
2020-04-26 17:35:53.894 EEST [32166] FATAL: the database system is in recovery mode
2020-04-26 17:35:55.285 EEST [32167] FATAL: the database system is in recovery mode
2020-04-26 17:35:56.346 EEST [32168] FATAL: the database system is in recovery mode
2020-04-26 17:35:57.416 EEST [32169] FATAL: the database system is in recovery mode
2020-04-26 17:35:58.270 EEST [32164] LOG: redo done at 6FD/ED969B98
2020-04-26 17:35:58.466 EEST [32170] FATAL: the database system is in recovery mode
2020-04-26 17:35:59.555 EEST [32171] FATAL: the database system is in recovery mode
2020-04-26 17:36:00.606 EEST [32172] FATAL: the database system is in recovery mode
2020-04-26 17:36:02.746 EEST [32174] FATAL: the database system is in recovery mode
2020-04-26 17:36:03.409 EEST [8385] LOG: database system is ready to accept connections
2020-04-26 17:36:03.805 EEST [32175] FATAL: the database system is in recovery mode
Best regards,
Radu
Radu Radutiu <rradutiu@gmail.com> writes: > Can you guide me how to debug postgresql crash? A stack trace would be pretty useful. https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend regards, tom lane
Re: How to debug "background worker "parallel worker" (PID 32152) wasterminated by signal 11: Segmentation fault"
От
Mahendra Singh Thalor
Дата:
On Mon, 27 Apr 2020 at 17:52, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Radu Radutiu <rradutiu@gmail.com> writes: > > Can you guide me how to debug postgresql crash? > > A stack trace would be pretty useful. > > https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend > > regards, tom lane > > If you have given "-c" option while starting the server, then as your background worker(PID: 32152) crashed with segment fault so you should have core dump for this inside data folder. Check that core.32152 is exist inside your data folder or not. If exists, then fire "gdb postgres data/core.32152" and then "bt" to get stack trace. -- Thanks and Regards Mahendra Singh Thalor EnterpriseDB: http://www.enterprisedb.com