Re: pg_dump: error: schema with OID 7956828 does not exist

On Thu, Nov 07, 2019 at 11:59:24PM -0700, nemo wrote:
>Hello, I have a problem on PG12
>os:CentOS Linux release 7.4.1708 (Core)
>db:PostgreSQL 12.0
>
>the db log is
>2019-11-08 11:59:40.385 CST,,,19005,,5dc4e7d2.4a3d,1,,2019-11-08 11:58:10
>CST,1/0,0,WARNING,57P02,"terminating connection because of crash of another
>server process","The postmaster has commanded this server process to roll
>back the current transaction and exit, because another server process exited
>abnormally and possibly corrupted shared memory.","In a moment you should be
>able to reconnect to the database and repeat your command.",,,,,,"quickdie,
>postgres.c:2717","
>
>os message ,i find is not oom
>Nov  6 10:00:02 xx-xx crond: /usr/sbin/sendmail: Cannot allocate memory
>Nov  8 16:23:59 xx-xx kernel: Reserving 161MB of memory at 688MB for
>crashkernel (System RAM: 8191MB)
>Nov  8 16:23:59 xx-xx kernel: Early memory node ranges
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0x0009f000-0x0009ffff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000a0000-0x000effff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000f0000-0x000fffff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0xbffe0000-0xbfffffff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0xc0000000-0xfeffbfff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfeffc000-0xfeffffff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0xff000000-0xfffbffff]
>Nov  8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfffc0000-0xffffffff]
>Nov  8 16:23:59 xx-xx kernel: please try 'cgroup_disable=memory' option if
>you don't want memory cgroups
>Nov  8 16:23:59 xx-xx kernel: Initializing cgroup subsys memory
>Nov  8 16:23:59 xx-xx kernel: Freeing initrd memory: 17248k freed
>Nov  8 16:23:59 xx-xx kernel: Non-volatile memory driver v1.3
>Nov  8 16:23:59 xx-xx kernel: crash memory driver: version 1.1
>Nov  8 16:23:59 xx-xx kernel: Freeing unused kernel memory: 1764k freed
>Nov  8 16:24:00 xx-xx kernel: [TTM] Zone  kernel: Available graphics memory:
>4005228 kiB
>Nov  8 16:24:00 xx-xx kernel: [TTM] Zone   dma32: Available graphics memory:
>2097152 kiB
>Nov  8 17:36:48 xx-xx kernel: Reserving 161MB of memory at 688MB for
>crashkernel (System RAM: 8191MB)
>Nov  8 17:36:48 xx-xx kernel: Early memory node ranges
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0x0009f000-0x0009ffff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000a0000-0x000effff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000f0000-0x000fffff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0xbffe0000-0xbfffffff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0xc0000000-0xfeffbfff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfeffc000-0xfeffffff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0xff000000-0xfffbffff]
>Nov  8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfffc0000-0xffffffff]
>Nov  8 17:36:48 xx-xx kernel: please try 'cgroup_disable=memory' option if
>you don't want memory cgroups
>Nov  8 17:36:48 xx-xx kernel: Initializing cgroup subsys memory
>Nov  8 17:36:48 xx-xx kernel: Freeing initrd memory: 17824k freed
>Nov  8 17:36:48 xx-xx kernel: Non-volatile memory driver v1.3
>Nov  8 17:36:48 xx-xx kernel: crash memory driver: version 1.1
>Nov  8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 1984k freed
>Nov  8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 424k freed
>Nov  8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 556k freed
>Nov  8 17:36:49 xx-xx kernel: [TTM] Zone  kernel: Available graphics memory:
>4004518 kiB
>Nov  8 17:36:49 xx-xx kernel: [TTM] Zone   dma32: Available graphics memory:
>2097152 kiB
>Nov  8 17:38:12 xx-xx kernel: Reserving 161MB of memory at 688MB for
>crashkernel (System RAM: 8191MB)
>Nov  8 17:38:12 xx-xx kernel: Early memory node ranges
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x0009f000-0x0009ffff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000a0000-0x000effff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000f0000-0x000fffff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xbffe0000-0xbfffffff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xc0000000-0xfeffbfff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfeffc000-0xfeffffff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xff000000-0xfffbffff]
>Nov  8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfffc0000-0xffffffff]
>Nov  8 17:38:12 xx-xx kernel: please try 'cgroup_disable=memory' option if
>you don't want memory cgroups
>Nov  8 17:38:12 xx-xx kernel: Initializing cgroup subsys memory
>Nov  8 17:38:12 xx-xx kernel: Freeing initrd memory: 17824k freed
>Nov  8 17:38:12 xx-xx kernel: Non-volatile memory driver v1.3
>Nov  8 17:38:12 xx-xx kernel: crash memory driver: version 1.1
>Nov  8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 1984k freed
>Nov  8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 424k freed
>Nov  8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 556k freed
>Nov  8 17:38:13 xx-xx kernel: [TTM] Zone  kernel: Available graphics memory:
>4004514 kiB
>Nov  8 17:38:13 xx-xx kernel: [TTM] Zone   dma32: Available graphics memory:
>2097152 kiB
>Nov  8 18:35:12 xx-xx kernel: Reserving 161MB of memory at 688MB for
>crashkernel (System RAM: 8191MB)
>Nov  8 18:35:12 xx-xx kernel: Early memory node ranges
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x0009f000-0x0009ffff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000a0000-0x000effff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000f0000-0x000fffff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xbffe0000-0xbfffffff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xc0000000-0xfeffbfff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfeffc000-0xfeffffff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xff000000-0xfffbffff]
>Nov  8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfffc0000-0xffffffff]
>Nov  8 18:35:12 xx-xx kernel: please try 'cgroup_disable=memory' option if
>you don't want memory cgroups
>Nov  8 18:35:12 xx-xx kernel: Initializing cgroup subsys memory
>Nov  8 18:35:12 xx-xx kernel: Freeing initrd memory: 17824k freed
>Nov  8 18:35:12 xx-xx kernel: Non-volatile memory driver v1.3
>Nov  8 18:35:12 xx-xx kernel: crash memory driver: version 1.1
>Nov  8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 1984k freed
>Nov  8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 424k freed
>Nov  8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 556k freed
>Nov  8 18:35:13 xx-xx kernel: [TTM] Zone  kernel: Available graphics memory:
>4004514 kiB
>Nov  8 18:35:13 xx-xx kernel: [TTM] Zone   dma32: Available graphics memory:
>2097152 kiB
>Nov  8 18:47:14 xx-xx kernel: Reserving 161MB of memory at 688MB for
>crashkernel (System RAM: 8191MB)
>Nov  8 18:47:14 xx-xx kernel: Early memory node ranges
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0x0009f000-0x0009ffff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000a0000-0x000effff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0x000f0000-0x000fffff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0xbffe0000-0xbfffffff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0xc0000000-0xfeffbfff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfeffc000-0xfeffffff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0xff000000-0xfffbffff]
>Nov  8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem
>0xfffc0000-0xffffffff]
>Nov  8 18:47:14 xx-xx kernel: please try 'cgroup_disable=memory' option if
>you don't want memory cgroups
>Nov  8 18:47:14 xx-xx kernel: Initializing cgroup subsys memory
>Nov  8 18:47:14 xx-xx kernel: Freeing initrd memory: 17824k freed
>Nov  8 18:47:14 xx-xx kernel: Non-volatile memory driver v1.3
>Nov  8 18:47:14 xx-xx kernel: crash memory driver: version 1.1
>Nov  8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 1984k freed
>Nov  8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 424k freed
>Nov  8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 556k freed
>Nov  8 18:47:15 xx-xx kernel: [TTM] Zone  kernel: Available graphics memory:
>4004518 kiB
>Nov  8 18:47:15 xx-xx kernel: [TTM] Zone   dma32: Available graphics memory:
>2097152 kiB
>
>os error message:
>Nov  8 14:30:56 xx-xx kernel: postgres[26427]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>Nov  8 14:31:41 xx-xx kernel: postgres[26453]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>Nov  8 14:32:27 xx-xx kernel: postgres[26479]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>Nov  8 14:33:12 xx-xx kernel: postgres[26523]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>Nov  8 14:33:58 xx-xx kernel: postgres[26556]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>Nov  8 14:40:43 xx-xx kernel: postgres[26571]: segfault at 0 ip
>00007f0e0a9e8641 sp 00007fff971a9598 error 4 in
>libc-2.17.so[7f0e0a879000+1c3000]
>
>then i run the pg_dump
>$ pg_dump -Fc -d testdb -f test-20191108.dump
>pg_dump: error: schema with OID 7956828 does not exist
>
># select * from pg_namespace where oid = 7956828 ;
> oid | nspname | nspowner | nspacl
>-----+---------+----------+--------
>(0 rows)
>
>please help me!!!
>

Clearly the postgres process is crashing because of segfault. Please see
[1] (there's a section about core dumps) and give us backtrace and other
details.

Other than that, it'd be interesting to know where does the schema OID
come from, so maybe investigate a bit in the server log and catalogs and
try to figure out which object is referring to the OID.

[1] https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services