Обсуждение: BUG #7533: Client is not able to connect cascade standby incase basebackup is taken from hot standby

The following bug has been logged on the website:

Bug reference:      7533
Logged by:          Amit Kapila
Email address:      amit.kapila@huawei.com
PostgreSQL version: 9.2.0
Operating system:   Suse
Description:        =


M host is primary, S host is standby and CS host is cascaded standby. =


1.Set up postgresql-9.2beta2/RC1 on  all hosts. =

2.Execute command initdb on host M to create fresh database. =

3.Modify the configure file postgresql.conf on host M like this=EF=BC=9A =

     listen_addresses =3D 'M' =

   port =3D 15210 =

   wal_level =3D hot_standby   =

   max_wal_senders =3D 4 =

   hot_standby =3D on =

4.modify the configure file pg_hba.conf on host M like this=EF=BC=9A =

host     replication     repl             M/24            md5 =

5.Start the server on host M as primary. =

6.Connect one client to primary server and create a user =E2=80=98repl=E2=
=80=99 =

  Create user repl superuser password '123'; =

7.Use the command pg_basebackup on the host S to retrieve database of
primary host =

pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
-v -h M -p 15210 -U repl =E2=80=93W =

8. Copy one recovery.conf.sample from share folder of package to database
folder of the host S. Then rename this file to recovery.conf =

9.Modify the file recovery.conf on host S as below: =

             standby_mode =3D on =

             primary_conninfo =3D 'host=3DM port=3D15210 user=3Drepl passwo=
rd=3D123' =

10. Modify the file postgresql.conf on host S as follow: =

       listen_addresses =3D 'S' =

11.Start the server on host S as standby server. =

12.Use the command pg_basebackup on the host CS to retrieve database of
standby host =

pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
-v -h M -p 15210 -U repl =E2=80=93W =

13.Modify the file recovery.conf on host CS as below: =

   standby_mode =3D on =

   primary_conninfo =3D 'host=3DS port=3D15210 user=3Drepl password=3D123' =

14. Modify the file postgresql.conf on host S as follow: =

     listen_addresses =3D 'CS' =

15.Start the server on host CS as Cascaded standby server node. =

16. Try to connect a client to host CS but it gives error as: =

    FATAL:  the database system is starting up =



Observations related to bug
------------------------------
In the above scenario it is observed that Start-up process has read all data
(in our defect scenario minRecoveryPoint is 5016220) till the position
5016220 and then it goes and check for recovery consistency by following
condition in function CheckRecoveryConsistency: =

        if (!reachedConsistency && =

                XLByteLE(minRecoveryPoint, EndRecPtr) && =

                XLogRecPtrIsInvalid(ControlFile->backupStartPoint)) =


At this point first two conditions are true but last condition is not true
because still redo has not been applied and hence backupStartPoint has not
been reset. So it does not signal postmaster regarding consistent stage.
After this it goes and applies the redo and then reset backupStartPoint and
then it goes to read next set of record. Since all records have been already
read, so it starts waiting for the new record from the Standby node. But
since there is no new record from Standby node coming so it keeps waiting
for that and it does not get chance to recheck the recovery consistent
level. And hence client connection does not get allowed.

On Wed, Sep 12, 2012 at 8:47 PM,  <amit.kapila@huawei.com> wrote:
> The following bug has been logged on the website:
>
> Bug reference:      7533
> Logged by:          Amit Kapila
> Email address:      amit.kapila@huawei.com
> PostgreSQL version: 9.2.0
> Operating system:   Suse
> Description:
>
> M host is primary, S host is standby and CS host is cascaded standby.
>
> 1.Set up postgresql-9.2beta2/RC1 on  all hosts.
> 2.Execute command initdb on host M to create fresh database.
> 3.Modify the configure file postgresql.conf on host M like this：
>      listen_addresses = 'M'
>    port = 15210
>    wal_level = hot_standby
>    max_wal_senders = 4
>    hot_standby = on
> 4.modify the configure file pg_hba.conf on host M like this：
> host     replication     repl             M/24            md5
> 5.Start the server on host M as primary.
> 6.Connect one client to primary server and create a user ‘repl’
>   Create user repl superuser password '123';
> 7.Use the command pg_basebackup on the host S to retrieve database of
> primary host
> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
> -v -h M -p 15210 -U repl –W
> 8. Copy one recovery.conf.sample from share folder of package to database
> folder of the host S. Then rename this file to recovery.conf
> 9.Modify the file recovery.conf on host S as below:
>              standby_mode = on
>              primary_conninfo = 'host=M port=15210 user=repl password=123'
> 10. Modify the file postgresql.conf on host S as follow:
>        listen_addresses = 'S'
> 11.Start the server on host S as standby server.
> 12.Use the command pg_basebackup on the host CS to retrieve database of
> standby host
> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
> -v -h M -p 15210 -U repl –W
> 13.Modify the file recovery.conf on host CS as below:
>    standby_mode = on
>    primary_conninfo = 'host=S port=15210 user=repl password=123'
> 14. Modify the file postgresql.conf on host S as follow:
>      listen_addresses = 'CS'
> 15.Start the server on host CS as Cascaded standby server node.
> 16. Try to connect a client to host CS but it gives error as:
>     FATAL:  the database system is starting up

This procedures didn't reproduce the problem in HEAD. But when I restarted
the master server between the step 11 and 12, I was able to reproduce the
problem.

> Observations related to bug
> ------------------------------
> In the above scenario it is observed that Start-up process has read all data
> (in our defect scenario minRecoveryPoint is 5016220) till the position
> 5016220 and then it goes and check for recovery consistency by following
> condition in function CheckRecoveryConsistency:
>         if (!reachedConsistency &&
>                 XLByteLE(minRecoveryPoint, EndRecPtr) &&
>                 XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
>
> At this point first two conditions are true but last condition is not true
> because still redo has not been applied and hence backupStartPoint has not
> been reset. So it does not signal postmaster regarding consistent stage.
> After this it goes and applies the redo and then reset backupStartPoint and
> then it goes to read next set of record. Since all records have been already
> read, so it starts waiting for the new record from the Standby node. But
> since there is no new record from Standby node coming so it keeps waiting
> for that and it does not get chance to recheck the recovery consistent
> level. And hence client connection does not get allowed.

If cascaded standby starts a recovery at a normal checkpoint record,
this problem will not happen. Because if wal_level is set to hot_standby,
XLOG_RUNNING_XACTS WAL record always follows after the normal
checkpont record. So while XLOG_RUNNING_XACTS record is being replayed,
ControlFile->backupStartPoint can be reset, and then cascaded standby
can pass through the consistency test.

The problem happens when cascaded standby starts a recovery at a
shutdown checkpoint record. In this case, no WAL record might follow
the checkpoint one yet. So, after replaying the shutdown checkpoint
record, cascaded standby needs to wait for new WAL record to appear
before reaching the code block for resetting ControlFile->backupStartPoint.
The cascaded standby cannot reach a consistent state and a client cannot
connect to the cascaded standby until new WAL has arrived.

Attached patch will fix the problem. In this patch, if recovery is
beginning at a shutdown checkpoint record, any ControlFile fields
(like backupStartPoint) required for checking that an end-of-backup is
reached are not set at first. IOW, cascaded standby thinks that the
database is consistent from the beginning. This is safe because
a shutdown checkpoint record means that there is no running database
activity at that point and the database is in consistent state.

Comments? Review?

Regards,

--
Fujii Masao

On Thursday, September 13, 2012 12:34 AM Fujii Masao wrote:
On Wed, Sep 12, 2012 at 8:47 PM,  <amit.kapila@huawei.com> wrote:
>> The following bug has been logged on the website:
>
>> Bug reference:      7533
>> Logged by:          Amit Kapila
>> Email address:      amit.kapila@huawei.com
>> PostgreSQL version: 9.2.0
>> Operating system:   Suse
>> Description:
>
>> M host is primary, S host is standby and CS host is cascaded standby.
>


> This procedures didn't reproduce the problem in HEAD. But when I =
restarted the master server between the step 11 and=20
> 12, I was able to reproduce the problem.

  We also observed that it didn't appear in 9.2rc1 due to commit =
b8b69d89905e04b910bcd65efce1791477b45d35 by Tom.
  The reason is checkpoint WAL will come from master after above fix , =
and cascaded standby will not stuck in LOOP.
  However if we increase Checkpoint interval, this does appear, some =
times we need to try 4-5 times.

>> Observations related to bug
>> ------------------------------
>> In the above scenario it is observed that Start-up process has read=20
>> all data (in our defect scenario minRecoveryPoint is 5016220) till =
the=20
>> position 5016220 and then it goes and check for recovery consistency=20
>> by following condition in function CheckRecoveryConsistency:
>>         if (!reachedConsistency &&
>>                 XLByteLE(minRecoveryPoint, EndRecPtr) &&
>>                 XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
>>
>> At this point first two conditions are true but last condition is not =

>> true because still redo has not been applied and hence=20
>> backupStartPoint has not been reset. So it does not signal postmaster =
regarding consistent stage.
>> After this it goes and applies the redo and then reset=20
>> backupStartPoint and then it goes to read next set of record. Since=20
>> all records have been already read, so it starts waiting for the new=20
>> record from the Standby node. But since there is no new record from=20
>> Standby node coming so it keeps waiting for that and it does not get=20
>> chance to recheck the recovery consistent level. And hence client =
connection does not get allowed.

If cascaded standby starts a recovery at a normal checkpoint record, =
this problem will not happen. Because if wal_level is set to =
hot_standby, XLOG_RUNNING_XACTS WAL record always follows after the =
normal checkpont record. So while XLOG_RUNNING_XACTS record is being =
replayed,
ControlFile->backupStartPoint can be reset, and then cascaded standby
can pass through the consistency test.

> The problem happens when cascaded standby starts a recovery at a =
shutdown checkpoint record. In this case, no WAL=20
> record might follow the checkpoint one yet. So, after replaying the =
shutdown checkpoint record, cascaded standby needs > to wait for new WAL =
record to appear before reaching the code block for resetting =
ControlFile->backupStartPoint.
> The cascaded standby cannot reach a consistent state and a client =
cannot connect to the cascaded standby until new WAL > has arrived.

In the above scenario, we are not doing shutdown so how can shutdown =
checkpoint record can come.=20

Also for the normal checkpoint case I have done brief analysis:
I have observed in code that ControlFile->minRecoveryPoint is updated =
while replaying XLOG_BACKUP_END WAL.
On hot standby S, this means that ControlFile->minRecoveryPoint will =
point to a lsn after checkpoint record.

Now when start recovery on cascaded standby CS after basebackup from hot =
standby S, minRecoveryPoint should point to lsn after checkpoint record =
lsn, so it might create problem.


> Attached patch will fix the problem. In this patch, if recovery is =
beginning at a shutdown checkpoint record, any=20
> ControlFile fields (like backupStartPoint) required for checking that =
an end-of-backup is reached are not set at first. > IOW, cascaded =
standby thinks that the database is consistent from the beginning. This =
is safe because a shutdown=20
> checkpoint record means that there is no running database activity at =
that point and the database is in consistent=20
> state.

  I shall test with the patch.

With Regards,
Amit Kapila.

On 12.09.2012 22:03, Fujii Masao wrote:
> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
>> The following bug has been logged on the website:
>>
>> Bug reference:      7533
>> Logged by:          Amit Kapila
>> Email address:      amit.kapila@huawei.com
>> PostgreSQL version: 9.2.0
>> Operating system:   Suse
>> Description:
>>
>> M host is primary, S host is standby and CS host is cascaded standby.
>>
>> 1.Set up postgresql-9.2beta2/RC1 on  all hosts.
>> 2.Execute command initdb on host M to create fresh database.
>> 3.Modify the configure file postgresql.conf on host M like this：
>>       listen_addresses = 'M'
>>     port = 15210
>>     wal_level = hot_standby
>>     max_wal_senders = 4
>>     hot_standby = on
>> 4.modify the configure file pg_hba.conf on host M like this：
>> host     replication     repl             M/24            md5
>> 5.Start the server on host M as primary.
>> 6.Connect one client to primary server and create a user ‘repl’
>>    Create user repl superuser password '123';
>> 7.Use the command pg_basebackup on the host S to retrieve database of
>> primary host
>> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
>> -v -h M -p 15210 -U repl –W
>> 8. Copy one recovery.conf.sample from share folder of package to database
>> folder of the host S. Then rename this file to recovery.conf
>> 9.Modify the file recovery.conf on host S as below:
>>               standby_mode = on
>>               primary_conninfo = 'host=M port=15210 user=repl password=123'
>> 10. Modify the file postgresql.conf on host S as follow:
>>         listen_addresses = 'S'
>> 11.Start the server on host S as standby server.
>> 12.Use the command pg_basebackup on the host CS to retrieve database of
>> standby host
>> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup -P
>> -v -h M -p 15210 -U repl –W
>> 13.Modify the file recovery.conf on host CS as below:
>>     standby_mode = on
>>     primary_conninfo = 'host=S port=15210 user=repl password=123'
>> 14. Modify the file postgresql.conf on host S as follow:
>>       listen_addresses = 'CS'
>> 15.Start the server on host CS as Cascaded standby server node.
>> 16. Try to connect a client to host CS but it gives error as:
>>      FATAL:  the database system is starting up
>
> This procedures didn't reproduce the problem in HEAD. But when I restarted
> the master server between the step 11 and 12, I was able to reproduce the
> problem.
>
>> Observations related to bug
>> ------------------------------
>> In the above scenario it is observed that Start-up process has read all data
>> (in our defect scenario minRecoveryPoint is 5016220) till the position
>> 5016220 and then it goes and check for recovery consistency by following
>> condition in function CheckRecoveryConsistency:
>>          if (!reachedConsistency&&
>>                  XLByteLE(minRecoveryPoint, EndRecPtr)&&
>>                  XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
>>
>> At this point first two conditions are true but last condition is not true
>> because still redo has not been applied and hence backupStartPoint has not
>> been reset. So it does not signal postmaster regarding consistent stage.
>> After this it goes and applies the redo and then reset backupStartPoint and
>> then it goes to read next set of record. Since all records have been already
>> read, so it starts waiting for the new record from the Standby node. But
>> since there is no new record from Standby node coming so it keeps waiting
>> for that and it does not get chance to recheck the recovery consistent
>> level. And hence client connection does not get allowed.
>
> If cascaded standby starts a recovery at a normal checkpoint record,
> this problem will not happen. Because if wal_level is set to hot_standby,
> XLOG_RUNNING_XACTS WAL record always follows after the normal
> checkpont record. So while XLOG_RUNNING_XACTS record is being replayed,
> ControlFile->backupStartPoint can be reset, and then cascaded standby
> can pass through the consistency test.
>
> The problem happens when cascaded standby starts a recovery at a
> shutdown checkpoint record. In this case, no WAL record might follow
> the checkpoint one yet. So, after replaying the shutdown checkpoint
> record, cascaded standby needs to wait for new WAL record to appear
> before reaching the code block for resetting ControlFile->backupStartPoint.
> The cascaded standby cannot reach a consistent state and a client cannot
> connect to the cascaded standby until new WAL has arrived.
>
> Attached patch will fix the problem. In this patch, if recovery is
> beginning at a shutdown checkpoint record, any ControlFile fields
> (like backupStartPoint) required for checking that an end-of-backup is
> reached are not set at first. IOW, cascaded standby thinks that the
> database is consistent from the beginning. This is safe because
> a shutdown checkpoint record means that there is no running database
> activity at that point and the database is in consistent state.

Hmm, I think the CheckRecoveryConsistency() call in the redo loop is
misplaced. It's called after we got a record from ReadRecord, but
*before* replaying it (rm_redo). Even if replaying record X makes the
system consistent, we won't check and notice that until we have fetched
record X+1. In this particular test case, record X is a shutdown
checkpoint record, but it could as well be a running-xacts record, or
the record that reaches minRecoveryPoint.

Does the problem go away if you just move the CheckRecoveryConsistency()
call *after* rm_redo (attached)?

- Heikki

On Thursday, September 13, 2012 5:52 PM Heikki Linnakangas wrote:
On 12.09.2012 22:03, Fujii Masao wrote:
> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
>> The following bug has been logged on the website:
>>
>>> Bug reference:      7533
>>> Logged by:          Amit Kapila
>>> Email address:      amit.kapila@huawei.com
>>> PostgreSQL version: 9.2.0
>>> Operating system:   Suse
>>> Description:
>>
>>> M host is primary, S host is standby and CS host is cascaded =
standby.
>>

>Hmm, I think the CheckRecoveryConsistency() call in the redo loop is =
misplaced. It's called after we got a record from=20
> ReadRecord, but *before* replaying it (rm_redo). Even if replaying =
record X makes the system consistent, we won't check=20
> and notice that until we have fetched record X+1. In this particular =
test case, record X is a shutdown checkpoint=20
> record, but it could as well be a running-xacts record, or the record =
that reaches minRecoveryPoint.

> Does the problem go away if you just move the =
CheckRecoveryConsistency() call *after* rm_redo (attached)?

This will resolve the problem I have reported but moving down might =
create another problem as due to function
recoveryPauseHere(), the recovery might get paused and in current code a =
client might be able to connect even in that state as  =
CheckRecoveryConsistency() is done before that. However after suggested =
change it might happen that Client will not be able to connect.
If you see any problem in what I explain then can we think of calling =
CheckRecoveryConsistency() both at current place and the place you have =
proposes. If my description doesn't make any sense (as that is only my =
suspicion) then we can move the function down and fix the defect.

Thank you for giving so quick response about the defect.

With Regards,
Amit Kapila.

On Thu, Sep 13, 2012 at 9:21 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote=
:
> On 12.09.2012 22:03, Fujii Masao wrote:
>>
>> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
>>>
>>> The following bug has been logged on the website:
>>>
>>> Bug reference:      7533
>>> Logged by:          Amit Kapila
>>> Email address:      amit.kapila@huawei.com
>>> PostgreSQL version: 9.2.0
>>> Operating system:   Suse
>>> Description:
>>>
>>> M host is primary, S host is standby and CS host is cascaded standby.
>>>
>>> 1.Set up postgresql-9.2beta2/RC1 on  all hosts.
>>> 2.Execute command initdb on host M to create fresh database.
>>> 3.Modify the configure file postgresql.conf on host M like this=EF=BC=
=9A
>>>       listen_addresses =3D 'M'
>>>     port =3D 15210
>>>     wal_level =3D hot_standby
>>>     max_wal_senders =3D 4
>>>     hot_standby =3D on
>>> 4.modify the configure file pg_hba.conf on host M like this=EF=BC=9A
>>> host     replication     repl             M/24            md5
>>> 5.Start the server on host M as primary.
>>> 6.Connect one client to primary server and create a user =E2=80=98repl=
=E2=80=99
>>>    Create user repl superuser password '123';
>>> 7.Use the command pg_basebackup on the host S to retrieve database of
>>> primary host
>>> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup
>>> -P
>>> -v -h M -p 15210 -U repl =E2=80=93W
>>> 8. Copy one recovery.conf.sample from share folder of package to databa=
se
>>> folder of the host S. Then rename this file to recovery.conf
>>> 9.Modify the file recovery.conf on host S as below:
>>>               standby_mode =3D on
>>>               primary_conninfo =3D 'host=3DM port=3D15210 user=3Drepl
>>> password=3D123'
>>> 10. Modify the file postgresql.conf on host S as follow:
>>>         listen_addresses =3D 'S'
>>> 11.Start the server on host S as standby server.
>>> 12.Use the command pg_basebackup on the host CS to retrieve database of
>>> standby host
>>> pg_basebackup  -D /opt/t38917/data -F p -x fetch -c fast -l repl_backup
>>> -P
>>> -v -h M -p 15210 -U repl =E2=80=93W
>>> 13.Modify the file recovery.conf on host CS as below:
>>>     standby_mode =3D on
>>>     primary_conninfo =3D 'host=3DS port=3D15210 user=3Drepl password=3D=
123'
>>> 14. Modify the file postgresql.conf on host S as follow:
>>>       listen_addresses =3D 'CS'
>>> 15.Start the server on host CS as Cascaded standby server node.
>>> 16. Try to connect a client to host CS but it gives error as:
>>>      FATAL:  the database system is starting up
>>
>>
>> This procedures didn't reproduce the problem in HEAD. But when I restart=
ed
>> the master server between the step 11 and 12, I was able to reproduce th=
e
>> problem.
>>
>>> Observations related to bug
>>> ------------------------------
>>> In the above scenario it is observed that Start-up process has read all
>>> data
>>> (in our defect scenario minRecoveryPoint is 5016220) till the position
>>> 5016220 and then it goes and check for recovery consistency by followin=
g
>>> condition in function CheckRecoveryConsistency:
>>>          if (!reachedConsistency&&
>>>                  XLByteLE(minRecoveryPoint, EndRecPtr)&&
>>>                  XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
>>>
>>> At this point first two conditions are true but last condition is not
>>> true
>>> because still redo has not been applied and hence backupStartPoint has
>>> not
>>> been reset. So it does not signal postmaster regarding consistent stage=
.
>>> After this it goes and applies the redo and then reset backupStartPoint
>>> and
>>> then it goes to read next set of record. Since all records have been
>>> already
>>> read, so it starts waiting for the new record from the Standby node. Bu=
t
>>> since there is no new record from Standby node coming so it keeps waiti=
ng
>>> for that and it does not get chance to recheck the recovery consistent
>>> level. And hence client connection does not get allowed.
>>
>>
>> If cascaded standby starts a recovery at a normal checkpoint record,
>> this problem will not happen. Because if wal_level is set to hot_standby=
,
>> XLOG_RUNNING_XACTS WAL record always follows after the normal
>> checkpont record. So while XLOG_RUNNING_XACTS record is being replayed,
>> ControlFile->backupStartPoint can be reset, and then cascaded standby
>> can pass through the consistency test.
>>
>> The problem happens when cascaded standby starts a recovery at a
>> shutdown checkpoint record. In this case, no WAL record might follow
>> the checkpoint one yet. So, after replaying the shutdown checkpoint
>> record, cascaded standby needs to wait for new WAL record to appear
>> before reaching the code block for resetting
>> ControlFile->backupStartPoint.
>> The cascaded standby cannot reach a consistent state and a client cannot
>> connect to the cascaded standby until new WAL has arrived.
>>
>> Attached patch will fix the problem. In this patch, if recovery is
>> beginning at a shutdown checkpoint record, any ControlFile fields
>> (like backupStartPoint) required for checking that an end-of-backup is
>> reached are not set at first. IOW, cascaded standby thinks that the
>> database is consistent from the beginning. This is safe because
>> a shutdown checkpoint record means that there is no running database
>> activity at that point and the database is in consistent state.
>
>
> Hmm, I think the CheckRecoveryConsistency() call in the redo loop is
> misplaced. It's called after we got a record from ReadRecord, but *before=
*
> replaying it (rm_redo). Even if replaying record X makes the system
> consistent, we won't check and notice that until we have fetched record X=
+1.
> In this particular test case, record X is a shutdown checkpoint record, b=
ut
> it could as well be a running-xacts record, or the record that reaches
> minRecoveryPoint.
>
> Does the problem go away if you just move the CheckRecoveryConsistency()
> call *after* rm_redo (attached)?

No, at least in my case. When recovery starts at shutdown checkpoint record=
 and
there is no record following the shutdown checkpoint, recovery gets in
wait state
before entering the main redo apply loop. That is, recovery starts waiting =
for
new WAL record to arrive, in ReadRecord just before the redo loop. So movin=
g
the CheckRecoveryConsistency() call after rm_redo cannot fix the problem wh=
ich
I reported. To fix the problem, we need to make the recovery reach the
consistent
point before the redo loop, i.e., in the CheckRecoveryConsistency()
just before the
redo loop.

Regards,

--=20
Fujii Masao

On Thursday, September 13, 2012 10:32 PM Fujii Masao wrote:
On Thu, Sep 13, 2012 at 9:21 PM, Heikki Linnakangas <hlinnaka@iki.fi> =
wrote:
> On 12.09.2012 22:03, Fujii Masao wrote:
>>
>> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
>>>
>>> The following bug has been logged on the website:
>>>
>>> Bug reference:      7533
>>> Logged by:          Amit Kapila
>>> Email address:      amit.kapila@huawei.com
>>> PostgreSQL version: 9.2.0
>>> Operating system:   Suse
>>> Description:
>>>
>>> M host is primary, S host is standby and CS host is cascaded =
standby.
>>>
>


>> Hmm, I think the CheckRecoveryConsistency() call in the redo loop is
>> misplaced. It's called after we got a record from ReadRecord, but =
*before*
>> replaying it (rm_redo). Even if replaying record X makes the system
>> consistent, we won't check and notice that until we have fetched =
record X+1.
>> In this particular test case, record X is a shutdown checkpoint =
record, but
>> it could as well be a running-xacts record, or the record that =
reaches
>> minRecoveryPoint.
>
>> Does the problem go away if you just move the =
CheckRecoveryConsistency()
>> call *after* rm_redo (attached)?

> No, at least in my case. When recovery starts at shutdown checkpoint =
record and
> there is no record following the shutdown checkpoint, recovery gets in
> wait state
> before entering the main redo apply loop. That is, recovery starts =
waiting for
> new WAL record to arrive, in ReadRecord just before the redo loop. So =
moving
> the CheckRecoveryConsistency() call after rm_redo cannot fix the =
problem which
>I reported. To fix the problem, we need to make the recovery reach the
> consistent
> point before the redo loop, i.e., in the CheckRecoveryConsistency()
> just before the redo loop.

I think may be in that case we need both the fixes, as the problem I =
have reported can be fixed with Heikki's patch.

With Regards,
Amit Kapila.

On Fri, Sep 14, 2012 at 12:21 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Thursday, September 13, 2012 10:32 PM Fujii Masao wrote:
> On Thu, Sep 13, 2012 at 9:21 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> On 12.09.2012 22:03, Fujii Masao wrote:
>>>
>>> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
>>>>
>>>> The following bug has been logged on the website:
>>>>
>>>> Bug reference:      7533
>>>> Logged by:          Amit Kapila
>>>> Email address:      amit.kapila@huawei.com
>>>> PostgreSQL version: 9.2.0
>>>> Operating system:   Suse
>>>> Description:
>>>>
>>>> M host is primary, S host is standby and CS host is cascaded standby.
>>>>
>>
>
>
>>> Hmm, I think the CheckRecoveryConsistency() call in the redo loop is
>>> misplaced. It's called after we got a record from ReadRecord, but *before*
>>> replaying it (rm_redo). Even if replaying record X makes the system
>>> consistent, we won't check and notice that until we have fetched record X+1.
>>> In this particular test case, record X is a shutdown checkpoint record, but
>>> it could as well be a running-xacts record, or the record that reaches
>>> minRecoveryPoint.
>>
>>> Does the problem go away if you just move the CheckRecoveryConsistency()
>>> call *after* rm_redo (attached)?
>
>> No, at least in my case. When recovery starts at shutdown checkpoint record and
>> there is no record following the shutdown checkpoint, recovery gets in
>> wait state
>> before entering the main redo apply loop. That is, recovery starts waiting for
>> new WAL record to arrive, in ReadRecord just before the redo loop. So moving
>> the CheckRecoveryConsistency() call after rm_redo cannot fix the problem which
>>I reported. To fix the problem, we need to make the recovery reach the
>> consistent
>> point before the redo loop, i.e., in the CheckRecoveryConsistency()
>> just before the redo loop.
>
> I think may be in that case we need both the fixes, as the problem I have reported can be fixed with Heikki's patch.

Agreed. And we should just add the CheckRecoveryConsistency() call after rm_redo
rather than moving it, as you suggested upthread.

Regards,

--
Fujii Masao

On Saturday, September 15, 2012 11:41 AM Fujii Masao wrote:
> On Fri, Sep 14, 2012 at 12:21 PM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Thursday, September 13, 2012 10:32 PM Fujii Masao wrote:
> > On Thu, Sep 13, 2012 at 9:21 PM, Heikki Linnakangas <hlinnaka@iki.fi>
> wrote:
> >> On 12.09.2012 22:03, Fujii Masao wrote:
> >>>
> >>> On Wed, Sep 12, 2012 at 8:47 PM,<amit.kapila@huawei.com>  wrote:
> >>>>
> >>>> The following bug has been logged on the website:
> >>>>
> >>>> Bug reference:      7533
> >>>> Logged by:          Amit Kapila
> >>>> Email address:      amit.kapila@huawei.com
> >>>> PostgreSQL version: 9.2.0
> >>>> Operating system:   Suse
> >>>> Description:
> >>>>
> >>>> M host is primary, S host is standby and CS host is cascaded
> standby.
> >>>>
> >>
> >
> >

> >
> >> No, at least in my case. When recovery starts at shutdown checkpoint
> record and
> >> there is no record following the shutdown checkpoint, recovery gets
> in
> >> wait state
> >> before entering the main redo apply loop. That is, recovery starts
> waiting for
> >> new WAL record to arrive, in ReadRecord just before the redo loop.
> So moving
> >> the CheckRecoveryConsistency() call after rm_redo cannot fix the
> problem which
> >>I reported. To fix the problem, we need to make the recovery reach
> the
> >> consistent
> >> point before the redo loop, i.e., in the CheckRecoveryConsistency()
> >> just before the redo loop.
> >
> > I think may be in that case we need both the fixes, as the problem I
> have reported can be fixed with Heikki's patch.
>
> Agreed. And we should just add the CheckRecoveryConsistency() call
> after rm_redo
> rather than moving it, as you suggested upthread.

If anybody doesn't have objection, then can we fix this in code. Basically
have the patch suggested by
Heikki with some modifications and Fujii's patch as fix of this defect and
if it's agreed upon, I can
combine both the fixes and prepare a single patch as fix of this defect.

With Regards,
Amit Kapila.

On Tue, Sep 25, 2012 at 5:35 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> If anybody doesn't have objection, then can we fix this in code. Basically
> have the patch suggested by
> Heikki with some modifications and Fujii's patch as fix of this defect and
> if it's agreed upon, I can
> combine both the fixes and prepare a single patch as fix of this defect.

Go for it!

Regards,

--
Fujii Masao