Re: START_REPLICATION silently truncates an overlong LSN component

Поиск
Список
Период
Сортировка
Искать

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Дата:
Вложения:
Hi,

Thanks for the report.

We introduced pg_parse_lsn() function for such issues.

It seems we should add it here to,  attached patch does that.
[Added a TAP test in patch,  but I dont think it needs to be committed]

I wonder if we should make the same change in libpqwalreceiver too.

Regards,
Ayush

RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Dear Ayush,

> It seems we should add it here to,  attached patch does that.
> [Added a TAP test in patch,  but I dont think it needs to be committed]

LGTM. I also think the test is not needed.

> I wonder if we should make the same change in libpqwalreceiver too.

I feel it should be, see attached. It could be combined with your patch.
I grepped the source with the keyword "scanf\(.*%X/%08X", and I could
not find other places where pg_parse_lsn() is usable as-is.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Jan Nidzwetzki <jan@planetscale.com>
Дата:
Вложения:
Hi Ayush, Hayato,

Thanks for the patches. I looked into them, and both apply cleanly to
master, and check-world passes. So, these changes look good to me.


On 11.09.26 04:35, Hayato Kuroda (Fujitsu) wrote:

>> I wonder if we should make the same change in libpqwalreceiver too.
> 
> I feel it should be, see attached. It could be combined with your patch.
> I grepped the source with the keyword "scanf\(.*%X/%08X", and I could
> not find other places where pg_parse_lsn() is usable as-is.


While looking for other places that need the same change, I found that
readTimeLineHistory() in src/backend/access/transam/timeline.c still
parses the switchpoint with sscanf("%X/%08X"). f31d6fbc31d fixed the
frontend copy of this code in pg_rewind, but not the backend one. The
attached 0003 patch converts it the same way and adds a TAP test to
004_timeline_switch.pl, including a case for the trailing characters
that are now rejected. As with the test in 0001, I'm not sure this one
needs to be committed.

In the attached v2 series, your patches are unchanged; I only added the
new 0003.

Best regards,
   Jan

-- 
Jan Nidzwetzki
PlanetScale Postgres Core Team

RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Hi Jan,

> While looking for other places that need the same change, I found that
> readTimeLineHistory() in src/backend/access/transam/timeline.c still
> parses the switchpoint with sscanf("%X/%08X"). f31d6fbc31d fixed the
> frontend copy of this code in pg_rewind, but not the backend one. The
> attached 0003 patch converts it the same way and adds a TAP test to
> 004_timeline_switch.pl, including a case for the trailing characters
> that are now rejected. As with the test in 0001, I'm not sure this one
> needs to be committed.

I also checked the source, and found that read_backup_label() parses the %X/%08X
in fscanf(). Can it be also replaced?

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Tue, Oct 20, 2026 at 4:36 PM Hayato Kuroda (Fujitsu)
 wrote:
> > As with the test in 0001, I'm not sure this one
> > needs to be committed.

I think we can omit the test in 0003. We made the similar change in pg_rewind
in f31d6fbc31d without adding a test, and pg_parse_lsn() is already covered
through other callers.

Regarding 0001 and 0002 patches, they look good to me.
So barring any objections, I will commit them.

> I also checked the source, and found that read_backup_label() parses the %X/%08X
> in fscanf(). Can it be also replaced?

I think that's possible.

Regards,

-- 
Fujii Masao


RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Fujii-san,

Thanks for the review.

> I think we can omit the test in 0003. We made the similar change in pg_rewind
> in f31d6fbc31d without adding a test, and pg_parse_lsn() is already covered
> through other callers.

+1.

> > I also checked the source, and found that read_backup_label() parses
> the %X/%08X
> > in fscanf(). Can it be also replaced?
> 
> I think that's possible.

OK, PSA 0004. We can reduce the magic number from 0004 if we export
MAXPG_LSNLEN, but not done yet.

All patches are attached to run CI.
0001 and 0002 are not changed, tests were removed from 0003.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Tue, Oct 20, 2026 at 8:29 PM Hayato Kuroda (Fujitsu)
 wrote:
> OK, PSA 0004.

Thanks!

+       if (fscanf(lfp, "INCREMENTAL FROM LSN: %17s\n", lsn) > 0 ||
+               !pg_parse_lsn(lsn, &istartpoint))

When fscanf() succeeds, pg_parse_lsn() is skipped. If the line is absent,
it parses the previous CHECKPOINT LOCATION value instead. Isn't this
problematic?

Since we don't use this LSN, how about removing the pg_parse_lsn() check
and istartpoint and simply rejecting the backup if fscanf() returns > 0?

> We can reduce the magic number from 0004 if we export
> MAXPG_LSNLEN

+1

Regards,

-- 
Fujii Masao


RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Fujii-san,

> +       if (fscanf(lfp, "INCREMENTAL FROM LSN: %17s\n", lsn) > 0 ||
> +               !pg_parse_lsn(lsn, &istartpoint))
> 
> When fscanf() succeeds, pg_parse_lsn() is skipped. If the line is absent,
> it parses the previous CHECKPOINT LOCATION value instead. Isn't this
> problematic?
> 
> Since we don't use this LSN, how about removing the pg_parse_lsn() check
> and istartpoint and simply rejecting the backup if fscanf() returns > 0?

Thanks for the feedback. PSA the fixed version. MAXPG_LSNLEN was also
exported in the header.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Wed, Oct 21, 2026 at 6:57 PM Hayato Kuroda (Fujitsu)
 wrote:
> Thanks for the feedback. PSA the fixed version. MAXPG_LSNLEN was also
> exported in the header.

Thanks for updating the patch!

+ if (fscanf(lfp,
+    "CHECKPOINT LOCATION: %" CppAsString2(MAXPG_LSNLEN) "s%c", lsn,
+    &ch) != 2 ||
+ ch != '\n' ||
+ !pg_parse_lsn(lsn, checkPointLoc))

This seems to accept malformed LSN fields such as:

    CHECKPOINT LOCATION: 0/123junk

Here,  represents an actual NUL byte. fscanf("%17s") reads past it,
but pg_parse_lsn() sees only 0/123. The old code rejected this because %c
read the NUL instead of a newline. START WAL LOCATION seems to
have the same issue.

BTW, I'm thinking of committing 0001 and 0002 first, as I've reviewed them.

Regards,

-- 
Fujii Masao


Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Thu, Oct 22, 2026 at 4:06 PM Fujii Masao  wrote:
> BTW, I'm thinking of committing 0001 and 0002 first, as I've reviewed them.

Firstly I've pushed 0001, 0002, and 0003 patches. Thanks!

Regards,

-- 
Fujii Masao


RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Dear Fujii-san,

Thanks for pushing the first half!

> + if (fscanf(lfp,
> +    "CHECKPOINT LOCATION: %" CppAsString2(MAXPG_LSNLEN) "s%c", lsn,
> +    &ch) != 2 ||
> + ch != '\n' ||
> + !pg_parse_lsn(lsn, checkPointLoc))
> 
> This seems to accept malformed LSN fields such as:
> 
>     CHECKPOINT LOCATION: 0/123junk
> 
> Here,  represents an actual NUL byte. fscanf("%17s") reads past it,
> but pg_parse_lsn() sees only 0/123. The old code rejected this because %c
> read the NUL instead of a newline. START WAL LOCATION seems to
> have the same issue.

Right, it should be rejected. One my primitive fix idea is to directly specify the
acceptable strings, like [0-9A-Fa-f/]. PSA the patch.
I also attached the test code to check invalid strings can be rejected, see 0002.
I'm not intended to be pushed.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Fri, Oct 23, 2026 at 12:24 PM Hayato Kuroda (Fujitsu)
 wrote:
> Right, it should be rejected. One my primitive fix idea is to directly specify the
> acceptable strings, like [0-9A-Fa-f/]. PSA the patch.

Thanks for updating the patch!

+ if (fscanf(lfp,
+    "START WAL LOCATION: %" CppAsString2(MAXPG_LSNLEN) "[0-9A-Fa-f/]
(file %08X%16s)%c",

For portability, isn't it better to use [0123456789ABCDEFabcdef/] instead of
[0-9A-Fa-f/]? pg_parse_lsn() also lists the hexadecimal digits explicitly.

- if (fscanf(lfp, "INCREMENTAL FROM LSN: %X/%08X\n", &hi, &lo) > 0)
+ if (fscanf(lfp, "INCREMENTAL FROM LSN: %" CppAsString2(MAXPG_LSNLEN)
"[0-9A-Fa-f/]\n",
+    lsn) > 0)

This change seems to allow a backup label containing "INCREMENTAL FROM
LSN: +0/1"
to bypass the check. Previously, %X accepted the leading sign, causing fscanf()
to return a positive value and the backup to be rejected. With this scanset,
fscanf() returns zero instead.

Since we only need to detect an incremental backup here and don't use
the LSN value, how about using "%s\n" instead of "[0-9A-Fa-f/]\n"? This would
preserve the rejection of the above input while also rejecting malformed
values such as "INCREMENTAL FROM LSN: junk".

Regards,


-- 
Fujii Masao


RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Fujii-san,

Thanks for reviewing!

> 
> + if (fscanf(lfp,
> +    "START WAL LOCATION: %" CppAsString2(MAXPG_LSNLEN)
> "[0-9A-Fa-f/]
> (file %08X%16s)%c",
> 
> For portability, isn't it better to use [0123456789ABCDEFabcdef/] instead of
> [0-9A-Fa-f/]? pg_parse_lsn() also lists the hexadecimal digits explicitly.

Done. I checked the C standard and the behavior when - is in the bracket seemed
to be the implementation-defined.

> Since we only need to detect an incremental backup here and don't use
> the LSN value, how about using "%s\n" instead of "[0-9A-Fa-f/]\n"? This would
> preserve the rejection of the above input while also rejecting malformed
> values such as "INCREMENTAL FROM LSN: junk".

Indeed, but %s cannot be used as-is because we need to pass the fixed-length char
array. If we do not have to take care the format of string, isn't it enough to
only check the prefix? I implemented like that in v6.

Best regards,
Hayato Kuroda
FUJITSU LIMITED 

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Jan Nidzwetzki <jan@planetscale.com>
Дата:
Hi Kuroda-san, Fujii-san,

Thanks for updating the patch set. I tested the two remaining ones on my
system. They apply cleanly and check-world passes.

Patch 0001
==========

I have just a small comment on the moved MAXPG_LSNLEN. It's currently a
hard-coded 17. It could make sense to export MAXPG_LSNCOMPONENT as well
and define MAXPG_LSNLEN as (2 * MAXPG_LSNCOMPONENT + 1), which would
document where the value comes from.

Patch 0002
==========

Kuroda-san wrote in the v5 mail:

> I also attached the test code to check invalid strings can be
rejected, see 0002.
> I'm not intended to be pushed.

I think this test is worth committing. As far as I can see, nothing else
covers the "this is an incremental backup, not a data directory" error,
and the new LSN checks would otherwise remain untested. However, we test
only an overlong high component. It could make sense to add this test
for the low component as well (e.g., using 0/123456789).

Best regards
  Jan

-- 
Jan Nidzwetzki
PlanetScale Postgres Core Team



RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Jan,

> Patch 0001
> ==========
> 
> I have just a small comment on the moved MAXPG_LSNLEN. It's currently a
> hard-coded 17. It could make sense to export MAXPG_LSNCOMPONENT as well
> and define MAXPG_LSNLEN as (2 * MAXPG_LSNCOMPONENT + 1), which
> would
> document where the value comes from.

I feel the notation cannot be used. CppAsString2() converts the argument to the
string, but it does not evaluate arithmetic at that time. So your suggestion
would be "%(2 * 8 + 1)s" and it won't work well. You can find that the compiler
raises warnings.

I feel we can add a static assert to ensure they remain consistent. I put in
pg_parse_lsn.c and it will be checked at compilation time. See new 0002 patch.
I separated them because it may be the independent improvement.

> Patch 0002
> ==========
> 
> Kuroda-san wrote in the v5 mail:
> 
> > I also attached the test code to check invalid strings can be
> rejected, see 0002.
> > I'm not intended to be pushed.
> 
> I think this test is worth committing. As far as I can see, nothing else
> covers the "this is an incremental backup, not a data directory" error,
> and the new LSN checks would otherwise remain untested. However, we test
> only an overlong high component. It could make sense to add this test
> for the low component as well (e.g., using 0/123456789).

Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:"
is nor verified, right? I still feel we can follow the manner.


Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Wed, Oct 28, 2026 at 12:18 PM Hayato Kuroda (Fujitsu)
 wrote:
> Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:"
> is nor verified, right? I still feel we can follow the manner.

I'm not convinced that 0002 and 0003 are necessary. OTOH, there seems to be
general agreement on 0001, so barring any objections, I'll commit that patch.

Regards,

-- 
Fujii Masao


Re: START_REPLICATION silently truncates an overlong LSN component

От:
Jan Nidzwetzki <jan@planetscale.com>
Дата:
Hi Kuroda-san, Fujii-san,

On 28.10.26 11:50, Fujii Masao wrote:
> On Wed, Oct 28, 2026 at 12:18 PM Hayato Kuroda (Fujitsu)
>  wrote:
>> Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:"
>> is nor verified, right? I still feel we can follow the manner.
> 
> I'm not convinced that 0002 and 0003 are necessary. OTOH, there seems to be
> general agreement on 0001, so barring any objections, I'll commit that patch.

No objections from my side.

Kuroda-san, you are right about CppAsString2(). I missed that the value
is stringified for scanf. Thanks for checking. So, I am fine with 0001.
The static assert and the test can be left out.

Best regards
   Jan

-- 
Jan Nidzwetzki
PlanetScale Postgres Core Team



RE: START_REPLICATION silently truncates an overlong LSN component

От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Hi Jan. Fujii-san,

Thanks for the review. I think 0002 is not a mandatory, can retain for now.
Someone can propose later in another thread.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Re: START_REPLICATION silently truncates an overlong LSN component

От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Thu, Oct 29, 2026 at 11:12 AM Hayato Kuroda (Fujitsu)
 wrote:
>
> Hi Jan. Fujii-san,
>
> Thanks for the review. I think 0002 is not a mandatory, can retain for now.
> Someone can propose later in another thread.

I've pushed the 0001 patch. Thanks!

Regards,

-- 
Fujii Masao


FAQ