BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

Поиск
Список
Период
Сортировка
Искать

BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

От:
PG Bug reporting form <noreply@postgresql.org>
Дата:
The following bug has been logged on the website:

Bug reference:      19897
Logged by:          Haruka Takatsuka
Email address:      harukat@sraoss.co.jp
PostgreSQL version: 15.0
Operating system:   Linux
Description:        

When using the following procedure to recover a base backup taken from a
standby,
the service would sometimes fail to start up with log messages in the
following strange sequence:

  LOG:  consistent recovery state reached at ...
  LOG:  invalid record length at ...
  FATAL:  recovery ended before configured recovery target was reached
  LOG:  database system is ready to accept read-only connections
  LOG:  startup process (PID 1234) exited with exit code 1

The base backup had the same result after repeated attempts.

(recovery steps)
$ rm -f data/15.5reco/log/*
$ rm data/15.5reco/standby.signal
$ touch data/15.5reco/recovery.signal
$ cat >> data/15.5reco/postgresql.conf <

Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

От:
Srinath Reddy Sadipiralla <srinath2133@gmail.com>
Дата:
Вложения:
Hi,

Thanks for the report and the analysis. I can reproduce this on master.

On Fri, Nov 13, 2026 at 2:27 PM PG Bug reporting form <noreply@postgresql.org> wrote:

diff --git a/src/backend/access/transam/xlogrecovery.c
b/src/backend/access/transam/xlogrecovery.c
index 52ac6d58423..796226efad4 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record)
                return true;
        }

+       /* Check if we should stop as soon as reaching consistency */
+       if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
+       {
+               ereport(LOG,
+                               (errmsg("recovery stopping after reaching
consistency")));
+
+               recoveryStopAfter = true;
+               recoveryStopXid = InvalidTransactionId;
+               recoveryStopTime = 0;
+               recoveryStopLSN = InvalidXLogRecPtr;
+               recoveryStopName[0] = '\0';
+               return true;
+       }
+
        if (rmid != RM_XACT_ID)
                return false;

@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
                }
        }

-       /* Check if we should stop as soon as reaching consistency */
-       if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
-       {
-               ereport(LOG,
-                               (errmsg("recovery stopping after reaching
consistency")));
-
-               recoveryStopAfter = true;
-               recoveryStopXid = InvalidTransactionId;
-               recoveryStopTime = 0;
-               recoveryStopLSN = InvalidXLogRecPtr;
-               recoveryStopName[0] = '\0';
-               return true;
-       }
-
        return false;
 }

Your proposed fix of moving the check above the rmgr test does fix the
reported case. But if the record that makes us consistent is a commit
or abort, it now returns before the SetLatestXTime() call, so the "last
completed transaction" time and pg_last_xact_replay_timestamp() would
miss that transaction. The attached 0001 instead goes back to the
pre-4f1b890b137 structure: the rmgr test becomes part of the condition
for the transaction-record block, so the SetLatestXTime still happens and
the immediate check is reached for every record. If this patch makes sense
i will write a TAP test, thoughts?


--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

От:
Zsolt Parragi <zsolt.parragi@percona.com>
Дата:
Вложения:
Hello

I can also confirm the reported case, and also that the attached patch fixes it.

But I also found a few similar cases, that could be fixed together with it.

The first was pointed out by claude:

Take a data directory without backup_label after a crash (such as
pg_ctl stop -m immediate), create recovery.signal and set:

recovery_target = 'immediate'
restore_command = 'false'
recovery_target_action = 'pause'

And then even with 0001, we see:

LOG:  starting point-in-time recovery to earliest consistent point
LOG:  redo starts at 0/0175BA40
LOG:  invalid record length at 0/0177B8B8: expected at least 24, got 0
LOG:  consistent recovery state reached at 0/0177B8B8
LOG:  redo done at 0/0177B720 ...
FATAL:  recovery ended before configured recovery target was reached

This scenario is explicitly mentioned in InitWalRecovery ("Or if you
just kill a running primary server and put it into archive recovery by
creating a recovery signal file"), but doesn't work, even with 0001
applied. An alternative patch fixes this and the original report both:
adding a check right after the main redo loop, before the
recovery_target_action switch.

And then there's a third issue, if instead of immediate, we use fast
in the above repro, which again doesn't work. That needs some
additional changes, and to avoid code duplication, I also did some
refactorings around it.

Additionally, all the above was done on the primary, and we can do the
same tests on a standby, where things are different. One case (backup
on a standby + backup ending in a restore point) requires your fix
from v1, so I think even through the reported case is also fixed by my
version, we need both to cover everything.

Attached is a 2 part series, where 0001 is the same as before, while
0002 adds additional changes and test coverage. We need both patches
for all tests to pass.

Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

От:
Zsolt Parragi <zsolt.parragi@percona.com>
Дата:
Вложения:
And after thinking about these changes, I realized that both of our
patches broke standby promotion: both resulted in cases where we
promoted the standby when we shouldn't have.

The attached v3 now doesn't modify the logic when we have a primary
connection set, and contains both patches in one file instead.

Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

От:
TAKATSUKA Haruka <harukat@sraoss.co.jp>
Дата:
Thank you for creating the patch.

I applied the contents of this v3 patch to version 15.x
(with no special adjustments except for pg_fallthrough.)

I tested it on the pgdata where the issue was initially detected in 15.x
and confirmed that the problem has been resolved.

I also verified that the regular regression tests passed on 15.x.
But TAP test No.9 of 003_recovery_targets.pl which is commented as
"Check behavior when recovery ends before target is reached" is failed.
Is is expected "FATAL:  recovery ended before configured recovery target was reached",
but after applying this patch the server now starts successfully.

+++ tap check in src/test/recovery +++
t/003_recovery_targets.pl .. 9/?
#   Failed test 'recovery end before target reached is a fatal error'
#   at t/003_recovery_targets.pl line 182.
t/003_recovery_targets.pl .. 29/? # Looks like you failed 1 test of 29.
t/003_recovery_targets.pl .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/29 subtests

Test Summary Report
-------------------
t/003_recovery_targets.pl (Wstat: 256 (exited 1) Tests: 29 Failed: 1)
  Failed test:  9
  Non-zero exit status: 1
Files=1, Tests=29, 202 wallclock secs ( 0.01 usr  0.01 sys +  2.01 cusr  3.90 csys =  5.93 CPU)
Result: FAIL
make: *** [Makefile:23: check] Error 1

$ tail tmp_check/log/003_recovery_targets_standby_8.log
2026-11-19 09:44:18.606 JST [6254] LOG:  starting point-in-time recovery to "does_not_exist"
2026-11-19 09:44:18.609 JST [6254] LOG:  restored log file "000000010000000000000003" from archive
2026-11-19 09:44:18.621 JST [6254] LOG:  redo starts at 0/3000028
2026-11-19 09:44:18.625 JST [6254] LOG:  restored log file "000000010000000000000004" from archive
2026-11-19 09:44:18.626 JST [6254] LOG:  consistent recovery state reached at 0/3000100
2026-11-19 09:44:18.626 JST [6251] LOG:  database system is ready to accept read-only connections
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
2026-11-19 09:44:18.635 JST [6254] LOG:  pausing at the end of recovery
2026-11-19 09:44:18.635 JST [6254] HINT:  Execute pg_wal_replay_resume() to promote.

I also tried on REL_15_STABLE head code and got same results.

with best regards,
Takatsuka Haruka / SRA OSS


On Sun, 15 Nov 2026 10:15:02 -0600
Zsolt Parragi  wrote:

> And after thinking about these changes, I realized that both of our
> patches broke standby promotion: both resulted in cases where we
> promoted the standby when we shouldn't have.
> 
> The attached v3 now doesn't modify the logic when we have a primary
> connection set, and contains both patches in one file instead.



FAQ