Postmaster doesn't correctly handle crashes in PM_STARTUP state

Hi,

While testing something I made the checkpointer process intentionally crash as
soon as it started up.  The odd thing I observed on macOS is that we start a
*new* checkpointer before shutting down:

2023-07-29 14:32:39.241 PDT [65031] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-07-29 14:32:39.244 PDT [65031] DEBUG:  reaping dead processes
2023-07-29 14:32:39.244 PDT [65031] LOG:  checkpointer process (PID 65032) was terminated by signal 11: Segmentation
fault:11
 
2023-07-29 14:32:39.244 PDT [65031] LOG:  terminating any other active server processes
2023-07-29 14:32:39.244 PDT [65031] DEBUG:  sending SIGQUIT to process 65034
2023-07-29 14:32:39.245 PDT [65031] DEBUG:  sending SIGQUIT to process 65033
2023-07-29 14:32:39.245 PDT [65031] DEBUG:  reaping dead processes
2023-07-29 14:32:39.245 PDT [65035] LOG:  process 65035 taking over ProcSignal slot 126, but it's not empty
2023-07-29 14:32:39.245 PDT [65031] DEBUG:  reaping dead processes
2023-07-29 14:32:39.245 PDT [65031] LOG:  shutting down because restart_after_crash is off

Note that a new process (65035) is started after the crash has been
observed. I added logging to StartChildProcess(), and the process that's
started is another checkpointer.

I could not initially reproduce this on linux.

After a fair bit of confusion, I figured out the reason: On macOS it takes a
bit longer for the startup process to finish, which means we're still in
PM_STARTUP state when we see that crash, instead of PM_RECOVERY or PM_RUN or
...

The problem is that unfortunately HandleChildCrash() doesn't change pmState
when in PM_STARTUP:

    /* We now transit into a state of waiting for children to die */
    if (pmState == PM_RECOVERY ||
        pmState == PM_HOT_STANDBY ||
        pmState == PM_RUN ||
        pmState == PM_STOP_BACKENDS ||
        pmState == PM_SHUTDOWN)
        pmState = PM_WAIT_BACKENDS;

Once I figured that out, I put a sleep(1) in StartupProcessMain(), and the
problem reproduces on linux as well.

I haven't fully dug through the history, this looks to be a quite old problem.


Arguably we might also be missing PM_SHUTDOWN_2, but I can't really see a bad
consequence of that.

Greetings,

Andres Freund