Our PostgreSQL 15.2 instance running on Ubuntu 18.04 has crashed with this error:
2023-04-05 09:24:03.448 UTC [15227] ERROR: index "pg_class_oid_index" contains unexpected zero page at block 0
2023-04-05 09:24:03.448 UTC [15227] HINT: Please REINDEX it.
...
2023-04-05 13:05:25.018 UTC [15437] root@test_behavior_638162834106895162 FATAL: index "pg_class_oid_index" contains unexpected zero page at block 0
2023-04-05 13:05:25.018 UTC [15437] root@test_behavior_638162834106895162 HINT: Please REINDEX it.
... (same error for a few more DBs)
2023-04-05 13:05:25.144 UTC [16965] root@test_behavior_638162855458823077 FATAL: index "pg_class_oid_index" contains unexpected zero page at block 0
2023-04-05 13:05:25.144 UTC [16965] root@test_behavior_638162855458823077 HINT: Please REINDEX it.
...
2023-04-05 13:05:25.404 UTC [17309] root@test_behavior_638162881641031612 PANIC: could not open critical system index 2662
2023-04-05 13:05:25.405 UTC [9372] LOG: server process (PID 17309) was terminated by signal 6: Aborted
2023-04-05 13:05:25.405 UTC [9372] LOG: terminating any other active server processes
We had the same thing happened about a month ago on a different database on the same cluster. For a while PG actually ran OK as long as you didn't access that specific DB, but when trying to back up that DB with pg_dump it would crash every time. At that time one of the disks hosting the ZFS dataset with the PG data directory on it was reporting errors, so we thought it was likely due to that.
Unfortunately, before we could replace the disks, PG crashed completely and would not start again at all, so I had to rebuild the cluster from scratch and restore from pg_dump backups (still onto the old, bad disks). Once the disks were replaced (all of them) I just copied the data to them using zfs send | zfs receive and didn't bother restoring pg_dump backups again - which was perhaps foolish in hindsight.
Well, yesterday it happened again. The server still restarted OK, so I took fresh pg_dump backups of the databases we care about (which ran fine), rebuilt the cluster and restored the pg_dump backups again - now onto the new disks, which are not reporting any problems.
So while everything is up and running now this error has me rather concerned. Could the error we're seeing now have been caused by some corruption in the PG data that's been there for a month (so it could still be attributed to the bad disk), which should now be fixed by having restored from backups onto good disks? Could this be a PG bug? What can I do to figure out why this is happening and prevent it from happening again? Advice appreciated!