Re: START_REPLICATION silently truncates an overlong LSN component
Re: START_REPLICATION silently truncates an overlong LSN component
Hi,
Thanks for the report.
We introduced pg_parse_lsn() function for such issues.
It seems we should add it here to, attached patch does that.
[Added a TAP test in patch, but I dont think it needs to be committed]
I wonder if we should make the same change in libpqwalreceiver too.
Regards,
Ayush
RE: START_REPLICATION silently truncates an overlong LSN component
Dear Ayush, > It seems we should add it here to, attached patch does that. > [Added a TAP test in patch, but I dont think it needs to be committed] LGTM. I also think the test is not needed. > I wonder if we should make the same change in libpqwalreceiver too. I feel it should be, see attached. It could be combined with your patch. I grepped the source with the keyword "scanf\(.*%X/%08X", and I could not find other places where pg_parse_lsn() is usable as-is. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
Hi Ayush, Hayato,
Thanks for the patches. I looked into them, and both apply cleanly to
master, and check-world passes. So, these changes look good to me.
On 11.09.26 04:35, Hayato Kuroda (Fujitsu) wrote:
>> I wonder if we should make the same change in libpqwalreceiver too.
>
> I feel it should be, see attached. It could be combined with your patch.
> I grepped the source with the keyword "scanf\(.*%X/%08X", and I could
> not find other places where pg_parse_lsn() is usable as-is.
While looking for other places that need the same change, I found that
readTimeLineHistory() in src/backend/access/transam/timeline.c still
parses the switchpoint with sscanf("%X/%08X"). f31d6fbc31d fixed the
frontend copy of this code in pg_rewind, but not the backend one. The
attached 0003 patch converts it the same way and adds a TAP test to
004_timeline_switch.pl, including a case for the trailing characters
that are now rejected. As with the test in 0001, I'm not sure this one
needs to be committed.
In the attached v2 series, your patches are unchanged; I only added the
new 0003.
Best regards,
Jan
--
Jan Nidzwetzki
PlanetScale Postgres Core Team
RE: START_REPLICATION silently truncates an overlong LSN component
От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Hi Jan,
> While looking for other places that need the same change, I found that
> readTimeLineHistory() in src/backend/access/transam/timeline.c still
> parses the switchpoint with sscanf("%X/%08X"). f31d6fbc31d fixed the
> frontend copy of this code in pg_rewind, but not the backend one. The
> attached 0003 patch converts it the same way and adds a TAP test to
> 004_timeline_switch.pl, including a case for the trailing characters
> that are now rejected. As with the test in 0001, I'm not sure this one
> needs to be committed.
I also checked the source, and found that read_backup_label() parses the %X/%08X
in fscanf(). Can it be also replaced?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Tue, Oct 20, 2026 at 4:36 PM Hayato Kuroda (Fujitsu) wrote: > > As with the test in 0001, I'm not sure this one > > needs to be committed. I think we can omit the test in 0003. We made the similar change in pg_rewind in f31d6fbc31d without adding a test, and pg_parse_lsn() is already covered through other callers. Regarding 0001 and 0002 patches, they look good to me. So barring any objections, I will commit them. > I also checked the source, and found that read_backup_label() parses the %X/%08X > in fscanf(). Can it be also replaced? I think that's possible. Regards, -- Fujii Masao
RE: START_REPLICATION silently truncates an overlong LSN component
От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Fujii-san, Thanks for the review. > I think we can omit the test in 0003. We made the similar change in pg_rewind > in f31d6fbc31d without adding a test, and pg_parse_lsn() is already covered > through other callers. +1. > > I also checked the source, and found that read_backup_label() parses > the %X/%08X > > in fscanf(). Can it be also replaced? > > I think that's possible. OK, PSA 0004. We can reduce the magic number from 0004 if we export MAXPG_LSNLEN, but not done yet. All patches are attached to run CI. 0001 and 0002 are not changed, tests were removed from 0003. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Tue, Oct 20, 2026 at 8:29 PM Hayato Kuroda (Fujitsu) wrote: > OK, PSA 0004. Thanks! + if (fscanf(lfp, "INCREMENTAL FROM LSN: %17s\n", lsn) > 0 || + !pg_parse_lsn(lsn, &istartpoint)) When fscanf() succeeds, pg_parse_lsn() is skipped. If the line is absent, it parses the previous CHECKPOINT LOCATION value instead. Isn't this problematic? Since we don't use this LSN, how about removing the pg_parse_lsn() check and istartpoint and simply rejecting the backup if fscanf() returns > 0? > We can reduce the magic number from 0004 if we export > MAXPG_LSNLEN +1 Regards, -- Fujii Masao
RE: START_REPLICATION silently truncates an overlong LSN component
От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Вложения:
Hi Fujii-san, > + if (fscanf(lfp, "INCREMENTAL FROM LSN: %17s\n", lsn) > 0 || > + !pg_parse_lsn(lsn, &istartpoint)) > > When fscanf() succeeds, pg_parse_lsn() is skipped. If the line is absent, > it parses the previous CHECKPOINT LOCATION value instead. Isn't this > problematic? > > Since we don't use this LSN, how about removing the pg_parse_lsn() check > and istartpoint and simply rejecting the backup if fscanf() returns > 0? Thanks for the feedback. PSA the fixed version. MAXPG_LSNLEN was also exported in the header. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Wed, Oct 21, 2026 at 6:57 PM Hayato Kuroda (Fujitsu)
wrote:
> Thanks for the feedback. PSA the fixed version. MAXPG_LSNLEN was also
> exported in the header.
Thanks for updating the patch!
+ if (fscanf(lfp,
+ "CHECKPOINT LOCATION: %" CppAsString2(MAXPG_LSNLEN) "s%c", lsn,
+ &ch) != 2 ||
+ ch != '\n' ||
+ !pg_parse_lsn(lsn, checkPointLoc))
This seems to accept malformed LSN fields such as:
CHECKPOINT LOCATION: 0/123junk
Here, represents an actual NUL byte. fscanf("%17s") reads past it,
but pg_parse_lsn() sees only 0/123. The old code rejected this because %c
read the NUL instead of a newline. START WAL LOCATION seems to
have the same issue.
BTW, I'm thinking of committing 0001 and 0002 first, as I've reviewed them.
Regards,
--
Fujii Masao
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Thu, Oct 22, 2026 at 4:06 PM Fujii Masao wrote: > BTW, I'm thinking of committing 0001 and 0002 first, as I've reviewed them. Firstly I've pushed 0001, 0002, and 0003 patches. Thanks! Regards, -- Fujii Masao
RE: START_REPLICATION silently truncates an overlong LSN component
Dear Fujii-san,
Thanks for pushing the first half!
> + if (fscanf(lfp,
> + "CHECKPOINT LOCATION: %" CppAsString2(MAXPG_LSNLEN) "s%c", lsn,
> + &ch) != 2 ||
> + ch != '\n' ||
> + !pg_parse_lsn(lsn, checkPointLoc))
>
> This seems to accept malformed LSN fields such as:
>
> CHECKPOINT LOCATION: 0/123junk
>
> Here, represents an actual NUL byte. fscanf("%17s") reads past it,
> but pg_parse_lsn() sees only 0/123. The old code rejected this because %c
> read the NUL instead of a newline. START WAL LOCATION seems to
> have the same issue.
Right, it should be rejected. One my primitive fix idea is to directly specify the
acceptable strings, like [0-9A-Fa-f/]. PSA the patch.
I also attached the test code to check invalid strings can be rejected, see 0002.
I'm not intended to be pushed.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Fri, Oct 23, 2026 at 12:24 PM Hayato Kuroda (Fujitsu) wrote: > Right, it should be rejected. One my primitive fix idea is to directly specify the > acceptable strings, like [0-9A-Fa-f/]. PSA the patch. Thanks for updating the patch! + if (fscanf(lfp, + "START WAL LOCATION: %" CppAsString2(MAXPG_LSNLEN) "[0-9A-Fa-f/] (file %08X%16s)%c", For portability, isn't it better to use [0123456789ABCDEFabcdef/] instead of [0-9A-Fa-f/]? pg_parse_lsn() also lists the hexadecimal digits explicitly. - if (fscanf(lfp, "INCREMENTAL FROM LSN: %X/%08X\n", &hi, &lo) > 0) + if (fscanf(lfp, "INCREMENTAL FROM LSN: %" CppAsString2(MAXPG_LSNLEN) "[0-9A-Fa-f/]\n", + lsn) > 0) This change seems to allow a backup label containing "INCREMENTAL FROM LSN: +0/1" to bypass the check. Previously, %X accepted the leading sign, causing fscanf() to return a positive value and the backup to be rejected. With this scanset, fscanf() returns zero instead. Since we only need to detect an incremental backup here and don't use the LSN value, how about using "%s\n" instead of "[0-9A-Fa-f/]\n"? This would preserve the rejection of the above input while also rejecting malformed values such as "INCREMENTAL FROM LSN: junk". Regards, -- Fujii Masao
RE: START_REPLICATION silently truncates an overlong LSN component
Hi Fujii-san, Thanks for reviewing! > > + if (fscanf(lfp, > + "START WAL LOCATION: %" CppAsString2(MAXPG_LSNLEN) > "[0-9A-Fa-f/] > (file %08X%16s)%c", > > For portability, isn't it better to use [0123456789ABCDEFabcdef/] instead of > [0-9A-Fa-f/]? pg_parse_lsn() also lists the hexadecimal digits explicitly. Done. I checked the C standard and the behavior when - is in the bracket seemed to be the implementation-defined. > Since we only need to detect an incremental backup here and don't use > the LSN value, how about using "%s\n" instead of "[0-9A-Fa-f/]\n"? This would > preserve the rejection of the above input while also rejecting malformed > values such as "INCREMENTAL FROM LSN: junk". Indeed, but %s cannot be used as-is because we need to pass the fixed-length char array. If we do not have to take care the format of string, isn't it enough to only check the prefix? I implemented like that in v6. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Jan Nidzwetzki <jan@planetscale.com>
Дата:
Hi Kuroda-san, Fujii-san, Thanks for updating the patch set. I tested the two remaining ones on my system. They apply cleanly and check-world passes. Patch 0001 ========== I have just a small comment on the moved MAXPG_LSNLEN. It's currently a hard-coded 17. It could make sense to export MAXPG_LSNCOMPONENT as well and define MAXPG_LSNLEN as (2 * MAXPG_LSNCOMPONENT + 1), which would document where the value comes from. Patch 0002 ========== Kuroda-san wrote in the v5 mail: > I also attached the test code to check invalid strings can be rejected, see 0002. > I'm not intended to be pushed. I think this test is worth committing. As far as I can see, nothing else covers the "this is an incremental backup, not a data directory" error, and the new LSN checks would otherwise remain untested. However, we test only an overlong high component. It could make sense to add this test for the low component as well (e.g., using 0/123456789). Best regards Jan -- Jan Nidzwetzki PlanetScale Postgres Core Team
RE: START_REPLICATION silently truncates an overlong LSN component
Hi Jan, > Patch 0001 > ========== > > I have just a small comment on the moved MAXPG_LSNLEN. It's currently a > hard-coded 17. It could make sense to export MAXPG_LSNCOMPONENT as well > and define MAXPG_LSNLEN as (2 * MAXPG_LSNCOMPONENT + 1), which > would > document where the value comes from. I feel the notation cannot be used. CppAsString2() converts the argument to the string, but it does not evaluate arithmetic at that time. So your suggestion would be "%(2 * 8 + 1)s" and it won't work well. You can find that the compiler raises warnings. I feel we can add a static assert to ensure they remain consistent. I put in pg_parse_lsn.c and it will be checked at compilation time. See new 0002 patch. I separated them because it may be the independent improvement. > Patch 0002 > ========== > > Kuroda-san wrote in the v5 mail: > > > I also attached the test code to check invalid strings can be > rejected, see 0002. > > I'm not intended to be pushed. > > I think this test is worth committing. As far as I can see, nothing else > covers the "this is an incremental backup, not a data directory" error, > and the new LSN checks would otherwise remain untested. However, we test > only an overlong high component. It could make sense to add this test > for the low component as well (e.g., using 0/123456789). Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:" is nor verified, right? I still feel we can follow the manner. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Wed, Oct 28, 2026 at 12:18 PM Hayato Kuroda (Fujitsu) wrote: > Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:" > is nor verified, right? I still feel we can follow the manner. I'm not convinced that 0002 and 0003 are necessary. OTOH, there seems to be general agreement on 0001, so barring any objections, I'll commit that patch. Regards, -- Fujii Masao
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Jan Nidzwetzki <jan@planetscale.com>
Дата:
Hi Kuroda-san, Fujii-san, On 28.10.26 11:50, Fujii Masao wrote: > On Wed, Oct 28, 2026 at 12:18 PM Hayato Kuroda (Fujitsu) > wrote: >> Hmm. But 5ffb8c74 also did not have any tests and the error "syntax error in history file:" >> is nor verified, right? I still feel we can follow the manner. > > I'm not convinced that 0002 and 0003 are necessary. OTOH, there seems to be > general agreement on 0001, so barring any objections, I'll commit that patch. No objections from my side. Kuroda-san, you are right about CppAsString2(). I missed that the value is stringified for scanf. Thanks for checking. So, I am fine with 0001. The static assert and the test can be left out. Best regards Jan -- Jan Nidzwetzki PlanetScale Postgres Core Team
RE: START_REPLICATION silently truncates an overlong LSN component
От:
"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Дата:
Hi Jan. Fujii-san, Thanks for the review. I think 0002 is not a mandatory, can retain for now. Someone can propose later in another thread. Best regards, Hayato Kuroda FUJITSU LIMITED
Re: START_REPLICATION silently truncates an overlong LSN component
От:
Fujii Masao <masao.fujii@gmail.com>
Дата:
On Thu, Oct 29, 2026 at 11:12 AM Hayato Kuroda (Fujitsu) wrote: > > Hi Jan. Fujii-san, > > Thanks for the review. I think 0002 is not a mandatory, can retain for now. > Someone can propose later in another thread. I've pushed the 0001 patch. Thanks! Regards, -- Fujii Masao