BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
От:
PG Bug reporting form <noreply@postgresql.org>
Дата:
The following bug has been logged on the website: Bug reference: 19897 Logged by: Haruka Takatsuka Email address: harukat@sraoss.co.jp PostgreSQL version: 15.0 Operating system: Linux Description: When using the following procedure to recover a base backup taken from a standby, the service would sometimes fail to start up with log messages in the following strange sequence: LOG: consistent recovery state reached at ... LOG: invalid record length at ... FATAL: recovery ended before configured recovery target was reached LOG: database system is ready to accept read-only connections LOG: startup process (PID 1234) exited with exit code 1 The base backup had the same result after repeated attempts. (recovery steps) $ rm -f data/15.5reco/log/* $ rm data/15.5reco/standby.signal $ touch data/15.5reco/recovery.signal $ cat >> data/15.5reco/postgresql.conf <
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
Hi,
Thanks for the report and the analysis. I can reproduce this on master.
Thanks for the report and the analysis. I can reproduce this on master.
On Fri, Nov 13, 2026 at 2:27 PM PG Bug reporting form <noreply@postgresql.org> wrote:
diff --git a/src/backend/access/transam/xlogrecovery.c
b/src/backend/access/transam/xlogrecovery.c
index 52ac6d58423..796226efad4 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record)
return true;
}
+ /* Check if we should stop as soon as reaching consistency */
+ if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
+ {
+ ereport(LOG,
+ (errmsg("recovery stopping after reaching
consistency")));
+
+ recoveryStopAfter = true;
+ recoveryStopXid = InvalidTransactionId;
+ recoveryStopTime = 0;
+ recoveryStopLSN = InvalidXLogRecPtr;
+ recoveryStopName[0] = '\0';
+ return true;
+ }
+
if (rmid != RM_XACT_ID)
return false;
@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
}
}
- /* Check if we should stop as soon as reaching consistency */
- if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
- {
- ereport(LOG,
- (errmsg("recovery stopping after reaching
consistency")));
-
- recoveryStopAfter = true;
- recoveryStopXid = InvalidTransactionId;
- recoveryStopTime = 0;
- recoveryStopLSN = InvalidXLogRecPtr;
- recoveryStopName[0] = '\0';
- return true;
- }
-
return false;
}
Your proposed fix of moving the check above the rmgr test does fix the
reported case. But if the record that makes us consistent is a commit
or abort, it now returns before the SetLatestXTime() call, so the "last
completed transaction" time and pg_last_xact_replay_timestamp() would
miss that transaction. The attached 0001 instead goes back to the
pre-4f1b890b137 structure: the rmgr test becomes part of the condition
for the transaction-record block, so the SetLatestXTime still happens and
the immediate check is reached for every record. If this patch makes sense
i will write a TAP test, thoughts?
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
Hello
I can also confirm the reported case, and also that the attached patch fixes it.
But I also found a few similar cases, that could be fixed together with it.
The first was pointed out by claude:
Take a data directory without backup_label after a crash (such as
pg_ctl stop -m immediate), create recovery.signal and set:
recovery_target = 'immediate'
restore_command = 'false'
recovery_target_action = 'pause'
And then even with 0001, we see:
LOG: starting point-in-time recovery to earliest consistent point
LOG: redo starts at 0/0175BA40
LOG: invalid record length at 0/0177B8B8: expected at least 24, got 0
LOG: consistent recovery state reached at 0/0177B8B8
LOG: redo done at 0/0177B720 ...
FATAL: recovery ended before configured recovery target was reached
This scenario is explicitly mentioned in InitWalRecovery ("Or if you
just kill a running primary server and put it into archive recovery by
creating a recovery signal file"), but doesn't work, even with 0001
applied. An alternative patch fixes this and the original report both:
adding a check right after the main redo loop, before the
recovery_target_action switch.
And then there's a third issue, if instead of immediate, we use fast
in the above repro, which again doesn't work. That needs some
additional changes, and to avoid code duplication, I also did some
refactorings around it.
Additionally, all the above was done on the primary, and we can do the
same tests on a standby, where things are different. One case (backup
on a standby + backup ending in a restore point) requires your fix
from v1, so I think even through the reported case is also fixed by my
version, we need both to cover everything.
Attached is a 2 part series, where 0001 is the same as before, while
0002 adds additional changes and test coverage. We need both patches
for all tests to pass.
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
And after thinking about these changes, I realized that both of our patches broke standby promotion: both resulted in cases where we promoted the standby when we shouldn't have. The attached v3 now doesn't modify the logic when we have a primary connection set, and contains both patches in one file instead.
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
От:
TAKATSUKA Haruka <harukat@sraoss.co.jp>
Дата:
Thank you for creating the patch. I applied the contents of this v3 patch to version 15.x (with no special adjustments except for pg_fallthrough.) I tested it on the pgdata where the issue was initially detected in 15.x and confirmed that the problem has been resolved. I also verified that the regular regression tests passed on 15.x. But TAP test No.9 of 003_recovery_targets.pl which is commented as "Check behavior when recovery ends before target is reached" is failed. Is is expected "FATAL: recovery ended before configured recovery target was reached", but after applying this patch the server now starts successfully. +++ tap check in src/test/recovery +++ t/003_recovery_targets.pl .. 9/? # Failed test 'recovery end before target reached is a fatal error' # at t/003_recovery_targets.pl line 182. t/003_recovery_targets.pl .. 29/? # Looks like you failed 1 test of 29. t/003_recovery_targets.pl .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/29 subtests Test Summary Report ------------------- t/003_recovery_targets.pl (Wstat: 256 (exited 1) Tests: 29 Failed: 1) Failed test: 9 Non-zero exit status: 1 Files=1, Tests=29, 202 wallclock secs ( 0.01 usr 0.01 sys + 2.01 cusr 3.90 csys = 5.93 CPU) Result: FAIL make: *** [Makefile:23: check] Error 1 $ tail tmp_check/log/003_recovery_targets_standby_8.log 2026-11-19 09:44:18.606 JST [6254] LOG: starting point-in-time recovery to "does_not_exist" 2026-11-19 09:44:18.609 JST [6254] LOG: restored log file "000000010000000000000003" from archive 2026-11-19 09:44:18.621 JST [6254] LOG: redo starts at 0/3000028 2026-11-19 09:44:18.625 JST [6254] LOG: restored log file "000000010000000000000004" from archive 2026-11-19 09:44:18.626 JST [6254] LOG: consistent recovery state reached at 0/3000100 2026-11-19 09:44:18.626 JST [6251] LOG: database system is ready to accept read-only connections cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory 2026-11-19 09:44:18.635 JST [6254] LOG: pausing at the end of recovery 2026-11-19 09:44:18.635 JST [6254] HINT: Execute pg_wal_replay_resume() to promote. I also tried on REL_15_STABLE head code and got same results. with best regards, Takatsuka Haruka / SRA OSS On Sun, 15 Nov 2026 10:15:02 -0600 Zsolt Parragi wrote: > And after thinking about these changes, I realized that both of our > patches broke standby promotion: both resulted in cases where we > promoted the standby when we shouldn't have. > > The attached v3 now doesn't modify the logic when we have a primary > connection set, and contains both patches in one file instead.