Hello.
> Yes, that is correct.
Mmm. I believed that the log came from a single server run, since the
PID (I believe the [359], [357] are PID) did not change through the
log lines.
> 2022-08-05 18:50:13 UTC::@:[359]:LOG: creating missing WAL directory "pg_wal/archive_status"
This means that someone removes the content of pg_wal directory.
Removing some WAL files in pg_wal may lead to this symptom.
> 2022-08-05 18:50:13 UTC::@:[359]:LOG: entering standby mode
> recovering 00000002.history
> 2022-08-05 18:50:13 UTC::@:[359]:LOG: ### [S] @6/B8000198: abort=(0/0)0/0, miss=(0/0)0/0, SbyMode=0, SbyModeReq=1
> 2022-08-05 18:50:13 UTC::@:[359]:LOG: database system was not properly shut down; automatic recovery in progress
> 2022-08-05 18:50:13 UTC::@:[359]:LOG: redo starts at 6/B80000E8
>
> And a few hours later, is when we see a panic
So, it seems that the *standby* received the inconsistent WAL stream
(aborted-contrecord not followed by a overwriting-missing-contrecord)
from the primary. Thus the inconsistency happened on the primary, not
on the standby.
So... I'm still failing to draw the big picutre of what is happening
here.
Could you show us the server configuration (dbservers involved and
their roles (primary/standby..)), and the exact steps when you restart
the server after carsh?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center