Обсуждение: BUG #17954: Postgres startup fails with `could not locate a valid checkpoint record`

The following bug has been logged on the website:

Bug reference:      17954
Logged by:          Utkarsh Srivastava
Email address:      srivastavautkarsh8097@gmail.com
PostgreSQL version: 12.12
Operating system:   RHEL/Linux
Description:

Hi everyone,

Thank you for your time. We are running PostgreSQL 12.12 in a CRI-O
container on top of CephFS. A few days ago we noticed that DB startup was
failing with the following error:
```
2023-05-14 05:13:13.678 UTC [1] LOG:  received smart shutdown request
2023-05-14 05:13:36.692 UTC [1] LOG:  could not open file "postmaster.pid":
No such file or directory
2023-05-14 05:13:36.692 UTC [1] LOG:  performing immediate shutdown because
data directory lock file is invalid
2023-05-14 05:13:36.692 UTC [1] LOG:  received immediate shutdown request
2023-05-14 05:13:36.692 UTC [1] LOG:  could not open file "postmaster.pid":
No such file or directory
2023-05-14 05:13:36.692 UTC [261282] WARNING:  terminating connection
because of crash of another server process
2023-05-14 05:13:36.692 UTC [261282] DETAIL:  The postmaster has commanded
this server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.
2023-05-14 05:13:36.692 UTC [261282] HINT:  In a moment you should be able
to reconnect to the database and repeat your command.
< --- Trimmed repetition ---> 
2023-05-14 05:13:36.739 UTC [1] LOG:  database system is shut down
2023-05-14 05:13:37.723 UTC [24] LOG:  database system was shut down at
2023-05-14 05:13:17 UTC
2023-05-14 05:13:37.723 UTC [24] LOG:  invalid resource manager ID 101 at
9/8BF289E8
2023-05-14 05:13:37.723 UTC [24] LOG:  invalid primary checkpoint record
2023-05-14 05:13:37.723 UTC [24] PANIC:  could not locate a valid checkpoint
record
2023-05-14 05:13:39.961 UTC [22] LOG:  startup process (PID 24) was
terminated by signal 6: Aborted
2023-05-14 05:13:39.961 UTC [22] LOG:  aborting startup due to startup
process failure
2023-05-14 05:13:40.117 UTC [22] LOG:  database system is shut down
2023-05-14 05:14:06.726 UTC [24] LOG:  database system was shut down at
2023-05-14 05:13:17 UTC
2023-05-14 05:14:06.726 UTC [24] LOG:  invalid resource manager ID 101 at
9/8BF289E8
```

- What could be the root cause of this issue? 
- Is this a known issue (I did search the archives but couldn't find it
though)? If yes, is this fixed in a PG 13, 14, 15?

Thank you

On Thu, Jun 01, 2023 at 01:11:20PM +0000, PG Bug reporting form wrote:
> - What could be the root cause of this issue?
> - Is this a known issue (I did search the archives but couldn't find it
> though)? If yes, is this fixed in a PG 13, 14, 15?

Hard to say for sure, but it looks like your host has a few problems.
This part from your logs refers to something that should not happen,
to begin with:

> 2023-05-14 05:13:13.678 UTC [1] LOG:  received smart shutdown request
> 2023-05-14 05:13:36.692 UTC [1] LOG:  could not open file "postmaster.pid":
> No such file or directory
> 2023-05-14 05:13:36.692 UTC [1] LOG:  performing immediate shutdown because
> data directory lock file is invalid
> 2023-05-14 05:13:36.692 UTC [1] LOG:  received immediate shutdown request
> 2023-05-14 05:13:36.692 UTC [1] LOG:  could not open file "postmaster.pid":
> No such file or directory

This LOG would come from either AddToDataDirLockFile() or
RecheckDataDirLockFile().  Still, the third entry I am quoting refers
to a recheck of the PID file, meaning that the postmaster has bumped
into what looks like a corrupted PID file.
--
Michael