Re: psql: FATAL: the database system is starting up

On 6/1/19 5:32 PM, Tom K wrote:

> 
> 
> Trying what we did above but on the second node:
> 

Was this node the primary?

To me the below looks like there are replication slots set up that are 
failing. Not sure how to deal with this at the moment. You might try 
single-user mode:

https://www.postgresql.org/docs/10/app-postgres.html
Single-User Mode
...

and see if that at least gets the server started. This is a highly 
restricted so do not expect much usability.

> Removing the recovery.conf and restarting yields the following messages:
> 
> 2019-06-01 20:31:12.231 EDT [21910] LOG:  database system was 
> interrupted; last known up at 2019-04-28 06:06:24 EDT
> 2019-06-01 20:31:12.505 EDT [21910] LOG:  invalid record length at 
> 0/4C35CDF8: wanted 24, got 0
> 2019-06-01 20:31:12.505 EDT [21910] LOG:  invalid primary checkpoint record
> 2019-06-01 20:31:12.505 EDT [21910] LOG:  using previous checkpoint 
> record at 0/4C34EDA8
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  redo record is at 
> 0/4C34ED70; shutdown FALSE
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  next transaction ID: 
> 0:1409831; next OID: 237578
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  next MultiXactId: 48; next 
> MultiXactOffset: 174
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  oldest unfrozen transaction 
> ID: 549, in database 1
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  oldest MultiXactId: 1, in 
> database 1
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  commit timestamp Xid 
> oldest/newest: 0/0
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  transaction ID wrap limit is 
> 2147484196, limited by database with OID 1
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  MultiXactId wrap limit is 
> 2147483648, limited by database with OID 1
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  starting up replication slots
> 2019-06-01 20:31:12.505 EDT [21910] DEBUG:  starting up replication 
> origin progress state
> 2019-06-01 20:31:12.505 EDT [21910] PANIC:  replication checkpoint has 
> wrong magic 0 instead of 307747550
> 2019-06-01 20:31:12.506 EDT [21908] DEBUG:  reaping dead processes
> 2019-06-01 20:31:12.506 EDT [21908] LOG:  startup process (PID 21910) 
> was terminated by signal 6: Aborted
> 2019-06-01 20:31:12.506 EDT [21908] LOG:  aborting startup due to 
> startup process failure
> 2019-06-01 20:31:12.506 EDT [21908] DEBUG:  shmem_exit(1): 0 
> before_shmem_exit callbacks to make
> 2019-06-01 20:31:12.506 EDT [21908] DEBUG:  shmem_exit(1): 5 
> on_shmem_exit callbacks to make
> 2019-06-01 20:31:12.506 EDT [21908] DEBUG:  cleaning up dynamic shared 
> memory control segment with ID 976986759
> 2019-06-01 20:31:12.508 EDT [21908] DEBUG:  proc_exit(1): 2 callbacks to 
> make
> 2019-06-01 20:31:12.508 EDT [21908] LOG:  database system is shut down
> 2019-06-01 20:31:12.508 EDT [21908] DEBUG:  exit(1)
> 2019-06-01 20:31:12.508 EDT [21908] DEBUG:  shmem_exit(-1): 0 
> before_shmem_exit callbacks to make
> 2019-06-01 20:31:12.508 EDT [21908] DEBUG:  shmem_exit(-1): 0 
> on_shmem_exit callbacks to make
> 2019-06-01 20:31:12.508 EDT [21908] DEBUG:  proc_exit(-1): 0 callbacks 
> to make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  logger shutting down
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  shmem_exit(0): 0 
> before_shmem_exit callbacks to make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  shmem_exit(0): 0 
> on_shmem_exit callbacks to make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  proc_exit(0): 0 callbacks to 
> make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  exit(0)
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  shmem_exit(-1): 0 
> before_shmem_exit callbacks to make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  shmem_exit(-1): 0 
> on_shmem_exit callbacks to make
> 2019-06-01 20:31:12.510 EDT [21909] DEBUG:  proc_exit(-1): 0 callbacks 
> to make
> 
>             -- 
>             Adrian Klaver
>             adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com