Обсуждение: Streaming replication breakdown
Hi Postgresql team,
My streaming replication has broken down. A request to provide a solution to fix it.
Please find below details:
This is Ubuntu (18.04), Postgresql 13.7 version.
Log details:
2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,1,,2022-06-05 09:18:59 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,2,,2022-06-05 09:18:59 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
2022-06-05 09:19:05.008 UTC,,,14122,,629c7508.372a,1,,2022-06-05 09:19:04 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:19:05.009 UTC,,,14122,,629c7508.372a,2,,2022-06-05 09:19:04 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
2022-06-05 09:19:10.013 UTC,,,14123,,629c750d.372b,1,,2022-06-05 09:19:09 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.784 UTC,,,14018,,629c73be.36c2,5,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.785 UTC,,,14123,,629c750d.372b,2,,2022-06-05 09:19:09 UTC,,0,FATAL,57P01,"terminating walreceiver process due to administrator command",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.785 UTC,,,14018,,629c73be.36c2,6,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.786 UTC,,,14018,,629c73be.36c2,7,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:07.790 UTC,,,14018,,629c73be.36c2,8,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:12.796 UTC,,,14018,,629c73be.36c2,9,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:17.800 UTC,,,14018,,629c73be.36c2,10,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:22.805 UTC,,,14018,,629c73be.36c2,11,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:27.810 UTC,,,14018,,629c73be.36c2,12,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:32.815 UTC,,,14018,,629c73be.36c2,13,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:37.820 UTC,,,14018,,629c73be.36c2,14,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:42.825 UTC,,,14018,,629c73be.36c2,15,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:47.830 UTC,,,14018,,629c73be.36c2,16,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:52.834 UTC,,,14018,,629c73be.36c2,17,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:19:05.008 UTC,,,14122,,629c7508.372a,1,,2022-06-05 09:19:04 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:19:05.009 UTC,,,14122,,629c7508.372a,2,,2022-06-05 09:19:04 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
2022-06-05 09:19:10.013 UTC,,,14123,,629c750d.372b,1,,2022-06-05 09:19:09 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.784 UTC,,,14018,,629c73be.36c2,5,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.785 UTC,,,14123,,629c750d.372b,2,,2022-06-05 09:19:09 UTC,,0,FATAL,57P01,"terminating walreceiver process due to administrator command",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.785 UTC,,,14018,,629c73be.36c2,6,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.786 UTC,,,14018,,629c73be.36c2,7,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:07.790 UTC,,,14018,,629c73be.36c2,8,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:12.796 UTC,,,14018,,629c73be.36c2,9,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:17.800 UTC,,,14018,,629c73be.36c2,10,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:22.805 UTC,,,14018,,629c73be.36c2,11,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:27.810 UTC,,,14018,,629c73be.36c2,12,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:32.815 UTC,,,14018,,629c73be.36c2,13,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:37.820 UTC,,,14018,,629c73be.36c2,14,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:42.825 UTC,,,14018,,629c73be.36c2,15,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:47.830 UTC,,,14018,,629c73be.36c2,16,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:52.834 UTC,,,14018,,629c73be.36c2,17,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
Kind regards,
Arun Maddukuri
Have you verified the integrity of a WAL file using pg_waldump?
Regards,
Ninad Shah
On Mon, Jun 6, 2022 at 10:46 PM Arun Maddukuri <arun.maddukuri07@gmail.com> wrote:
Hi Postgresql team,My streaming replication has broken down. A request to provide a solution to fix it.Please find below details:This is Ubuntu (18.04), Postgresql 13.7 version.Log details:2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,1,,2022-06-05 09:18:59 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,2,,2022-06-05 09:18:59 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
2022-06-05 09:19:05.008 UTC,,,14122,,629c7508.372a,1,,2022-06-05 09:19:04 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:19:05.009 UTC,,,14122,,629c7508.372a,2,,2022-06-05 09:19:04 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
2022-06-05 09:19:10.013 UTC,,,14123,,629c750d.372b,1,,2022-06-05 09:19:09 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.784 UTC,,,14018,,629c73be.36c2,5,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.785 UTC,,,14123,,629c750d.372b,2,,2022-06-05 09:19:09 UTC,,0,FATAL,57P01,"terminating walreceiver process due to administrator command",,,,,,,,,"","walreceiver"
2022-06-05 09:20:02.785 UTC,,,14018,,629c73be.36c2,6,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:02.786 UTC,,,14018,,629c73be.36c2,7,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:07.790 UTC,,,14018,,629c73be.36c2,8,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:12.796 UTC,,,14018,,629c73be.36c2,9,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:17.800 UTC,,,14018,,629c73be.36c2,10,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:22.805 UTC,,,14018,,629c73be.36c2,11,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:27.810 UTC,,,14018,,629c73be.36c2,12,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:32.815 UTC,,,14018,,629c73be.36c2,13,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:37.820 UTC,,,14018,,629c73be.36c2,14,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:42.825 UTC,,,14018,,629c73be.36c2,15,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:47.830 UTC,,,14018,,629c73be.36c2,16,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"
2022-06-05 09:20:52.834 UTC,,,14018,,629c73be.36c2,17,,2022-06-05 09:13:34 UTC,1/0,0,LOG,00000,"invalid resource manager ID 255 at 0/A60000A0",,,,,,,,,"","startup"Kind regards,Arun Maddukuri
On Mon, Jun 6, 2022 at 1:15 PM Arun Maddukuri <arun.maddukuri07@gmail.com> wrote:
Hi Postgresql team,My streaming replication has broken down. A request to provide a solution to fix it.Please find below details:This is Ubuntu (18.04), Postgresql 13.7 version.Log details:2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,1,,2022-06-05 09:18:59 UTC,,0,LOG,00000,"started streaming WAL from primary at 0/A6000000 on timeline 1",,,,,,,,,"","walreceiver"2022-06-05 09:19:00.006 UTC,,,14121,,629c7503.3729,2,,2022-06-05 09:18:59 UTC,,0,FATAL,XX000,"could not receive data from WAL stream: ERROR: requested starting point 0/A6000000 is ahead of the WAL flush position of this server 0/A50000D8",,,,,,,,,"","walreceiver"
It looks like the master server got removed and replaced with an earlier version of itself, such as from a backup. Talk to however might have been in a position to make that happen.
You will probably need to recreate the replica to get it streaming again. But first save a copy of the current replica someplace safe, as it may have the only copy which exists of some of the data lost from the master durings it mysterious adventure.
Cheers,
Jeff