PG version 16.14 upgrade went recovery state

Поиск
Список
Период
Сортировка
Искать

PG version 16.14 upgrade went recovery state

От:
K P Manoj <kpmanojpg@gmail.com>
Дата:

K P Manoj <kpmanojpg@gmail.com>

12:24 PM (3 hours ago)
to pgsql-hackers, pgsql-bugs
Hi Team,

I am encountering an issue when upgrading our PostgreSQL database from minor version 16.10 to 16.14.

During the upgrade, the database enters an indefinite recovery mode across the master, standby, and multiple cascading replicas. Due to legacy requirements, we are utilizing streaming replication for this setup. Interestingly, when performing the same migration to version 16.13, the process completes successfully without any issues.

Could you please confirm if there are any known bugs regarding this behavior in version 16.14?

For reference, please find the details of our test steps and logs below.

1. Upgrade Attempt to Version 16.14 (Database Stays in Recovery Mode)

Version Check:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# /usr/local/pgsql/bin/psql --version
psql (PostgreSQL) 16.14

Process Status:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ps -ef|grep rec
postgres      48      43  1 15:28 ?        00:00:01 postgres: startup recovering 0000000100002499000000E4
root         166      58  0 15:30 pts/0    00:00:00 grep --color=auto rec

WAL Directory Details:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -l 0000000100002499000000E4
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:25 0000000100002499000000E4

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -lt|more
total 55001312
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000059
drwxrws--- 2 postgres postgres   225280 Jul  2 14:54 archive_status
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000058
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000057
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000056
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000055
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000054
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000053
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000052
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000051
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000050
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004F
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004E
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004D

PostgreSQL Logs:

2026-07-02 15:28:38.709 UTC [43]: [3-1] user=,db=  >LOG:  starting PostgreSQL 16.14 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-07-02 15:28:38.709 UTC [43]: [4-1] user=,db=  >LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-07-02 15:28:38.709 UTC [43]: [5-1] user=,db=  >LOG:  listening on IPv6 address "::", port 5432
2026-07-02 15:28:38.722 UTC [43]: [6-1] user=,db=  >LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2026-07-02 15:28:38.753 UTC [48]: [1-1] user=,db=  >LOG:  database system was interrupted while in recovery at log time 2026-07-02 14:22:09 UTC
2026-07-02 15:28:38.753 UTC [48]: [2-1] user=,db=  >HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
2026-07-02 15:28:38.979 UTC [48]: [3-1] user=,db=  >LOG:  entering standby mode
2026-07-02 15:28:38.993 UTC [48]: [4-1] user=,db=  >LOG:  redo starts at 2499/CFF32450
2026-07-02 15:28:47.282 UTC [49]: [1-1] user=127.0.0.1,db=postgres  >FATAL:  the database system is not yet accepting connections
2026-07-02 15:28:47.282 UTC [49]: [2-1] user=127.0.0.1,db=postgres  >DETAIL:  Consistent recovery state has not been yet reached.
2026-07-02 15:28:57.291 UTC [57]: [1-1] user=127.0.0.1,db=postgres  >FATAL:  the database system is not yet accepting connections
2026-07-02 15:28:57.291 UTC [57]: [2-1] user=127.0.0.1,db=postgres  >DETAIL:  Consistent recovery state has not been yet reached.

2. Successful Migration to Version 16.13 (Streaming Replication Working)

Version Check:

root@pgreplica-master-0:/# psql --version
psql (PostgreSQL) 16.13 (Debian 16.13-1.pgdg12+1)

Process Status (Walreceiver Streaming Active):

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ps -ef|grep rec
postgres      48      43 32 15:40 ?        00:00:20 postgres: startup recovering 000000010000249B0000002E
postgres      50      43  7 15:40 ?        00:00:03 postgres: walreceiver streaming 249B/2E660000
root         105      80  0 15:41 pts/0    00:00:00 grep rec

WAL File Check:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -lth 000000010000249B0000002E
-rw------- 1 postgres postgres 16M Jul  2 15:41 000000010000249B0000002E

Database Connection Verification:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# psql -U postgres -c " select version () "
                                                           version                                                              
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 16.13 (Debian 16.13-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
(1 row)

Please let me know if any further details or logs are required to investigate this issue.

Best regards,

Manoj K P

Re: PG version 16.14 upgrade went recovery state

От:
Laurenz Albe <laurenz.albe@cybertec.at>
Дата:
On Fri, 2026-07-03 at 15:25 +0530, K P Manoj wrote:
> I am encountering an issue when upgrading our PostgreSQL database from minor version 16.10 to 16.14.
> 
> During the upgrade, the database enters an indefinite recovery mode across the master, standby,
> and multiple cascading replicas. Due to legacy requirements, we are utilizing streaming replication
> for this setup. Interestingly, when performing the same migration to version 16.13, the process
> completes successfully without any issues.
> 
> Could you please confirm if there are any known bugs regarding this behavior in version 16.14?

Yes, this could be bug 19490:
https://postgr.es/m/flat/19490-9c59c6a583513b99%40postgresql.org

Best wait for 16.15 before updating.

Yours,
Laurenz Albe


Re: PG version 16.14 upgrade went recovery state

От:
K P Manoj <kpmanojpg@gmail.com>
Дата:
Dear Michael,

Thank you for your email.

My primary objective is to mitigate CVE-2026-6473, and the required fix is only available in version 16.14. Therefore, I will wait for the next minor release.

Regards,

Manoj K P

On Mon, Jul 6, 2026 at 9:32 PM Michael Banck <mbanck@gmx.net> wrote:
On Sun, Jul 05, 2026 at 03:06:48PM +0200, Laurenz Albe wrote:
> On Fri, 2026-07-03 at 15:25 +0530, K P Manoj wrote:
> > I am encountering an issue when upgrading our PostgreSQL database from minor version 16.10 to 16.14.
> >
> > During the upgrade, the database enters an indefinite recovery mode across the master, standby,
> > and multiple cascading replicas. Due to legacy requirements, we are utilizing streaming replication
> > for this setup. Interestingly, when performing the same migration to version 16.13, the process
> > completes successfully without any issues.
> >
> > Could you please confirm if there are any known bugs regarding this behavior in version 16.14?
>
> Yes, this could be bug 19490:
> https://postgr.es/m/flat/19490-9c59c6a583513b99%40postgresql.org
>
> Best wait for 16.15 before updating.

The alternatve is to do a two-step process, where you ugprade everything
to 16.13, then upgrade the standbys to 16.14. As far as I know, the
leader needs to be on 16.11 or earlier (while the standbys needs to be
on 16.4) to hit this bug, so updating it to 16.12 or 16.13 should
circumvent this as well.

I wrote a blog-post summarizing the issue here:
https://www.credativ.de/en/blog/postgresql-en/replication-deadlock-bug-in-current-postgres-releases-14-16/


Michael


FAQ