Обсуждение: Supporting non-deterministic collations with tailoring rules.

    Todd Lang wrote:

> When creating a collation, in
> https://github.com/postgres/postgres/blob/master/src/backend/utils/adt/pg_locale_icu.c#L461
> it is opening the collator with the tailoring rules supplied. However, it
> has hardcoded the strength level UCOL_DEFAULT_STRENGTH. This has the effect
> of ignoring the "deterministic=false" you may have specified in your CREATE
> COLLATION call.


This is related to BUG #18771 previously reported at [1],
where the reporter notes that passing UCOL_DEFAULT works
for him whereas UCOL_DEFAULT_STRENGTH does not.
It looks like a documentation bug in ICU [2]
It says:

  strength: The default collation strength; one of UCOL_PRIMARY,
  UCOL_SECONDARY, UCOL_TERTIARY, UCOL_IDENTICAL,UCOL_DEFAULT_STRENGTH
  - can be also set in the rules.

But UCOL_DEFAULT_STRENGTH is an alias for UCOL_TERTIARY.
U_COL_DEFAULT is what should normally be passed to not override the
collation strength.

Now, by "it works", it means that the strength expressed in the rule
(with rules = '[strength 1]' in the case of the OP) takes effect.
This syntax is described at [3] (see "Rule Syntax" column)


There is a second problem: when the strength is specified in
the locale and not specified in the rules (as you did), it would also
be expected to take effect. It does not appear to be the case,
as if the rules were resetting the collation settings.
As mentioned in the thread at [1], Peter Eisentraut has submitted
this as a bug [4], but there hasn't been any follow-up to it
in 2.5 years.

> If, instead of UCOL_DEFAULT_STRENGTH, the code understood the
> deterministic parameter and passed either UCOL_PRIMARY for
> "deterministic=true", and UCOL_SECONDARY for "deterministic=false",
> this would preserve the attempt to obtain case-insensitivity in the
> locale while simultaneously allowing tailoring as expected.

We can't hardcode that deterministic=false implies that the strength
is 2. deterministic=false only says that the collation can have equal
strings that are not binary-equal.

To me, the most plausible fix on the Postgres side would be to pass
UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached,
which lets the user specify the strength in the rule, as the OP did in [1].


[1]:
https://www.postgresql.org/message-id/flat/18771-98bb23e455b0f367%40postgresql.org
[2]:
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/ucol_8h.html#a0cb1ddd81f322ed24e389f208eb35c8a
[3]: https://www.unicode.org/reports/tr35/tr35-collation.html#Setting_Options
[4]: https://unicode-org.atlassian.net/browse/ICU-22456


Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/

Ah, somehow I missed your email on this.  This is, in fact, exactly what should happen.  The ICU folks are updating
theirdocumentation to reflect this with https://github.com/unicode-org/icu/pull/3684/files . 

Is this small change a reasonable thing to include given the update in guidance from the ICU team?

-----Original Message-----
From: Daniel Verite <daniel@manitou-mail.org>
Sent: Wednesday, September 24, 2025 6:17 AM
To: Todd Lang <Todd.Lang@D2L.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Supporting non-deterministic collations with tailoring rules.

CAUTION: This email originated from outside of D2L. Do not respond to, click links or open attachments unless you
recognizethe sender and know the content is safe. 


        Todd Lang wrote:

> When creating a collation, in
> https://gith/
> ub.com%2Fpostgres%2Fpostgres%2Fblob%2Fmaster%2Fsrc%2Fbackend%2Futils%2
> Fadt%2Fpg_locale_icu.c%23L461&data=05%7C02%7CTodd.Lang%40D2L.com%7Cb34
> 6f047ed7944ebe01408ddfb5391b2%7C74bbca6d410b45b39b512a6aa6477079%7C0%7
> C0%7C638943058554088325%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW
> UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D
> %7C0%7C%7C%7C&sdata=4%2F6%2BalfTMFrnAzjQBt4i9Qa1BUUWBpaUJnGz%2B8dvy1s%
> 3D&reserved=0 it is opening the collator with the tailoring rules
> supplied. However, it has hardcoded the strength level
> UCOL_DEFAULT_STRENGTH. This has the effect of ignoring the
> "deterministic=false" you may have specified in your CREATE COLLATION
> call.


This is related to BUG #18771 previously reported at [1], where the reporter notes that passing UCOL_DEFAULT works for
himwhereas UCOL_DEFAULT_STRENGTH does not. 
It looks like a documentation bug in ICU [2] It says:

  strength: The default collation strength; one of UCOL_PRIMARY,
  UCOL_SECONDARY, UCOL_TERTIARY, UCOL_IDENTICAL,UCOL_DEFAULT_STRENGTH
  - can be also set in the rules.

But UCOL_DEFAULT_STRENGTH is an alias for UCOL_TERTIARY.
U_COL_DEFAULT is what should normally be passed to not override the collation strength.

Now, by "it works", it means that the strength expressed in the rule (with rules = '[strength 1]' in the case of the
OP)takes effect. 
This syntax is described at [3] (see "Rule Syntax" column)


There is a second problem: when the strength is specified in the locale and not specified in the rules (as you did), it
wouldalso be expected to take effect. It does not appear to be the case, as if the rules were resetting the collation
settings.
As mentioned in the thread at [1], Peter Eisentraut has submitted this as a bug [4], but there hasn't been any
follow-upto it in 2.5 years. 

> If, instead of UCOL_DEFAULT_STRENGTH, the code understood the
> deterministic parameter and passed either UCOL_PRIMARY for
> "deterministic=true", and UCOL_SECONDARY for "deterministic=false",
> this would preserve the attempt to obtain case-insensitivity in the
> locale while simultaneously allowing tailoring as expected.

We can't hardcode that deterministic=false implies that the strength is 2. deterministic=false only says that the
collationcan have equal strings that are not binary-equal. 

To me, the most plausible fix on the Postgres side would be to pass UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in
theattached, which lets the user specify the strength in the rule, as the OP did in [1]. 


[1]:
https://www.postgresql.org/message-id/flat/18771-98bb23e455b0f367%40postgresql.org
[2]:
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/ucol_8h.html#a0cb1ddd81f322ed24e389f208eb35c8a
[3]: https://www.unicode.org/reports/tr35/tr35-collation.html#Setting_Options
[4]: https://unicode-org.atlassian.net/browse/ICU-22456


Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/

    Todd Lang wrote:

> The ICU folks are updating their documentation to reflect
> this with https://github.com/unicode-org/icu/pull/3684/files .
>
> Is this small change a reasonable thing to include given the update in
> guidance from the ICU team?

I think so. Added to the commitfest:

https://commitfest.postgresql.org/patch/6084/


Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/

On 24.09.25 12:17, Daniel Verite wrote:
> To me, the most plausible fix on the Postgres side would be to pass
> UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached,
> which lets the user specify the strength in the rule, as the OP did in [1].

With this change, I don't see that the bug reported in ICU-22456 is 
fixed.  See attached my test case.

What change of behavior are you expecting from your patch?  Should there 
be test cases?

[resent for the list]

       Peter Eisentraut wrote:

> > To me, the most plausible fix on the Postgres side would be to pass
> > UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached,
> > which lets the user specify the strength in the rule, as the OP did in [1].
>
> With this change, I don't see that the bug reported in ICU-22456 is
> fixed.  See attached my test case.
>
> What change of behavior are you expecting from your patch?  Should there
> be test cases?

Indeed it does not fix the behavior reported in ICU-22456
(=collation properties being "reset" when adding rules)
It fixes the fact that specificying the strength in the rule itself
will be taken into account instead of the strength being
forced to tertiary.

PFA a new patch with a specific test case added.

With regards to reports on pgsql-bugs, it fixes #18771
and the second part of #19425.
It does not fix #19045, which looks the same as ICU-22456
It also does not fix the first part of #19425 (which looks the
same as #19045). We cannot fix that in Postgres AFAIU.
However adding the strength or other collation properties in
the rules can be used as a workaround against the ICU-22456
issue, provided the attached change is made.


Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/

On 15.03.26 19:52, Daniel Verite wrote:
> [resent for the list]
> 
>         Peter Eisentraut wrote:
> 
>>> To me, the most plausible fix on the Postgres side would be to pass
>>> UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached,
>>> which lets the user specify the strength in the rule, as the OP did in [1].
>>
>> With this change, I don't see that the bug reported in ICU-22456 is
>> fixed.  See attached my test case.
>>
>> What change of behavior are you expecting from your patch?  Should there
>> be test cases?
> 
> Indeed it does not fix the behavior reported in ICU-22456
> (=collation properties being "reset" when adding rules)
> It fixes the fact that specificying the strength in the rule itself
> will be taken into account instead of the strength being
> forced to tertiary.
> 
> PFA a new patch with a specific test case added.
> 
> With regards to reports on pgsql-bugs, it fixes #18771
> and the second part of #19425.
> It does not fix #19045, which looks the same as ICU-22456
> It also does not fix the first part of #19425 (which looks the
> same as #19045). We cannot fix that in Postgres AFAIU.
> However adding the strength or other collation properties in
> the rules can be used as a workaround against the ICU-22456
> issue, provided the attached change is made.

Ok, committed.

Thoughts on backpatching?  Seems like a straightforward bug fix.

    Peter Eisentraut wrote:

> Ok, committed.
>
> Thoughts on backpatching?  Seems like a straightforward bug fix.

Thanks for pushing the patch!
I thought it was not a candidate for backpatching because it's
a collation change.
Now, to have a problem with that change in a minor release,
a user would need to have an ICU collation with a rule containing
"[strength N]", where N<>3, even though it was ineffective (so
either the user did not notice or they did but kept it
nonetheless). Users who reported the problem hoped for a backpatch,
but we cannot say there is zero risk with that change in a minor
release. Personally I'm +0.


Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/

On 18.03.26 19:21, Daniel Verite wrote:
>     Peter Eisentraut wrote:
> 
>> Ok, committed.
>>
>> Thoughts on backpatching?  Seems like a straightforward bug fix.
> 
> Thanks for pushing the patch!
> I thought it was not a candidate for backpatching because it's
> a collation change.
> Now, to have a problem with that change in a minor release,
> a user would need to have an ICU collation with a rule containing
> "[strength N]", where N<>3, even though it was ineffective (so
> either the user did not notice or they did but kept it
> nonetheless). Users who reported the problem hoped for a backpatch,
> but we cannot say there is zero risk with that change in a minor
> release. Personally I'm +0.

Ok, let's not then.