Обсуждение: fixes for the Danish locale

In Danish, the sequence 'aa' is sometimes treated as a single letter
which collates after 'z'.

Some regression tests got into 9.5, and are still in 9.6beta3, which
fail due to assuming they know how things will sort or compare.

I thought the easiest way to deal with it was just to change the test
data to use 'ab...' rather than 'aa...' to represent an
early-collating string.

With these applied, this now passes:

LANG=danish make check

Cheers,

Jeff

Jeff Janes <jeff.janes@gmail.com> writes:
> In Danish, the sequence 'aa' is sometimes treated as a single letter
> which collates after 'z'.
> Some regression tests got into 9.5, and are still in 9.6beta3, which
> fail due to assuming they know how things will sort or compare.

Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
members covering this ...
        regards, tom lane

On Thu, Jul 21, 2016 at 5:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
> members covering this ...

We're not going to have a build farm member for every locale the local
systems support.

Perhaps the build farm script should pick a random locale for each
run. Either a random locale from the set on the OS or a random
language from a list of locale that the regression tests are intended
to be safe for.

-- 
greg

On Thu, Jul 21, 2016 at 11:26 AM, Greg Stark <stark@mit.edu> wrote:
> Perhaps the build farm script should pick a random locale for each
> run. Either a random locale from the set on the OS or a random
> language from a list of locale that the regression tests are intended
> to be safe for.

That's more or less what I did with the amcheck regression tests.

-- 
Peter Geoghegan

Greg Stark <stark@mit.edu> writes:
> On Thu, Jul 21, 2016 at 5:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
>> members covering this ...

> We're not going to have a build farm member for every locale the local
> systems support.

Probably not, but Danish seems odd enough to be worth testing.  Aside
from this issue, I found one in the pltcl tests.

> Perhaps the build farm script should pick a random locale for each
> run. Either a random locale from the set on the OS or a random
> language from a list of locale that the regression tests are intended
> to be safe for.

Nah, we have a hard enough time with reproducibility of buildfarm results
without deliberately injecting transient failures.
        regards, tom lane

On Thu, Jul 21, 2016 at 11:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Perhaps the build farm script should pick a random locale for each
>> run. Either a random locale from the set on the OS or a random
>> language from a list of locale that the regression tests are intended
>> to be safe for.
>
> Nah, we have a hard enough time with reproducibility of buildfarm results
> without deliberately injecting transient failures.

It could be pseudo-random, and so deterministic per buildfarm animal.
That's what I did.

-- 
Peter Geoghegan

Peter Geoghegan <pg@heroku.com> writes:
> On Thu, Jul 21, 2016 at 11:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Nah, we have a hard enough time with reproducibility of buildfarm results
>> without deliberately injecting transient failures.

> It could be pseudo-random, and so deterministic per buildfarm animal.
> That's what I did.

I'm not impressed with that proposal either --- then we don't even have
any control over what set of locales are getting tested.

Note that there are certain locales we've deliberately chosen not to
support in some regression tests (see e.g. plpython_unicode.sql), so
I'm not really willing to buy into the idea that "any random locale found
on a buildfarm animal should work" anyway.  I'm much more interested in
supporting locales that someone cares enough about to configure a
buildfarm animal for.
        regards, tom lane

On Thu, Jul 21, 2016 at 9:44 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Jeff Janes <jeff.janes@gmail.com> writes:
>> In Danish, the sequence 'aa' is sometimes treated as a single letter
>> which collates after 'z'.
>> Some regression tests got into 9.5, and are still in 9.6beta3, which
>> fail due to assuming they know how things will sort or compare.
>
> Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
> members covering this ...
>

My CentOS box came with 735 locales installed, so testing all of them
on a regular basis would be quite a task.  And it doesn't help that
many of them seem to be very slow compared to C locale.

I guess the good news is that nothing I tested which was working in
9.5 is broken in 9.6, but several things which were working in 9.4 did
get broken in 9.5 and still are in 9.6.

The Danish fix will probably also fix the (very large) Norwegian family.

The Welsh (cy_GB) apparently put 'dd' after 'f', which breaks row
level security in much the same way as 'aa' does.

I think that that will cover all of the ones that were working in 9.4.

Does testing in other locales ever uncover bugs other than those in
the tests themselves?  Is it worth trying to maintain broad coverage?

Cheers,

Jeff

On Thu, Jul 21, 2016 at 11:44 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Note that there are certain locales we've deliberately chosen not to
> support in some regression tests (see e.g. plpython_unicode.sql), so
> I'm not really willing to buy into the idea that "any random locale found
> on a buildfarm animal should work" anyway.  I'm much more interested in
> supporting locales that someone cares enough about to configure a
> buildfarm animal for.

That seems like a high standard to me. Locale rules are known to
change, and are explicitly versioned by glibc, for example.

-- 
Peter Geoghegan

On Thu, Jul 21, 2016 at 11:49 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
> Does testing in other locales ever uncover bugs other than those in
> the tests themselves?  Is it worth trying to maintain broad coverage?

Potentially, yes. The strxfrm() inconsistency issue disproportionately
affected de_DE.utf8, for example. There were other locales that were
affected less severely, and I think the majority were not shown to be
affected at all.

That being said, it probably wouldn't have caught that particular
issue if we had broad coverage. It probably would catch a broken test,
though.

-- 
Peter Geoghegan

On 07/21/2016 02:26 PM, Greg Stark wrote:
> On Thu, Jul 21, 2016 at 5:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
>> members covering this ...
> We're not going to have a build farm member for every locale the local
> systems support.
>
> Perhaps the build farm script should pick a random locale for each
> run. Either a random locale from the set on the OS or a random
> language from a list of locale that the regression tests are intended
> to be safe for.
>


I don't see why we shouldn't have a buildfarm machine that tests a very 
large number of locales. It takes a very lightly resourced machine like 
nightjar just over two minutes per locale. The list of locales to test 
is a setting in the config file.

cheers

andrew

Jeff Janes <jeff.janes@gmail.com> writes:
> In Danish, the sequence 'aa' is sometimes treated as a single letter
> which collates after 'z'.
> Some regression tests got into 9.5, and are still in 9.6beta3, which
> fail due to assuming they know how things will sort or compare.

As of HEAD, "LANG=danish make check-world" passes for me, which it
did not before the round of fixes I just pushed.

I see that the core tests fall over in Turkish still :-(
        regards, tom lane

On Thu, Jul 21, 2016 at 2:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Jeff Janes <jeff.janes@gmail.com> writes:
>> In Danish, the sequence 'aa' is sometimes treated as a single letter
>> which collates after 'z'.
>> Some regression tests got into 9.5, and are still in 9.6beta3, which
>> fail due to assuming they know how things will sort or compare.
>
> As of HEAD, "LANG=danish make check-world" passes for me, which it
> did not before the round of fixes I just pushed.
>
> I see that the core tests fall over in Turkish still :-(

Turkish has never passed (at least back to 9.0).  It looks like it is
in the stemming functions.  I don't understand why, I would think
everything other than English would be failing those if the regression
tests hard-code English stemming expectations but fail to arrange for
English stemming rules.

Cheers,

Jeff

Jeff Janes <jeff.janes@gmail.com> writes:
> On Thu, Jul 21, 2016 at 2:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I see that the core tests fall over in Turkish still :-(

> Turkish has never passed (at least back to 9.0).  It looks like it is
> in the stemming functions.  I don't understand why, I would think
> everything other than English would be failing those if the regression
> tests hard-code English stemming expectations but fail to arrange for
> English stemming rules.

It looks to me like the 'simple' dictionary assumes it can apply the
lowercasing rules implied by LC_CTYPE regardless of which language
it's supposedly working on.  This is probably something we should
improve sometime, but I doubt it's an easy change.
        regards, tom lane

On Thu, Jul 21, 2016 at 11:49 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
> On Thu, Jul 21, 2016 at 9:44 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Jeff Janes <jeff.janes@gmail.com> writes:
>>> In Danish, the sequence 'aa' is sometimes treated as a single letter
>>> which collates after 'z'.
>>> Some regression tests got into 9.5, and are still in 9.6beta3, which
>>> fail due to assuming they know how things will sort or compare.
>>
>> Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
>> members covering this ...
>>
>
> My CentOS box came with 735 locales installed, so testing all of them
> on a regular basis would be quite a task.  And it doesn't help that
> many of them seem to be very slow compared to C locale.
>
> I guess the good news is that nothing I tested which was working in
> 9.5 is broken in 9.6, but several things which were working in 9.4 did
> get broken in 9.5 and still are in 9.6.
>
> The Danish fix will probably also fix the (very large) Norwegian family.
>
> The Welsh (cy_GB) apparently put 'dd' after 'f', which breaks row
> level security in much the same way as 'aa' does.
>
> I think that that will cover all of the ones that were working in 9.4.

The attached patch fixes regression tests for Welsh (cy_GB), needed in
9.5 and 9.6.

Cheers,

Jeff

Jeff Janes <jeff.janes@gmail.com> writes:
> The attached patch fixes regression tests for Welsh (cy_GB), needed in
> 9.5 and 9.6.

Pushed, thanks.
        regards, tom lane

On 07/22/2016 03:59 AM, Jeff Janes wrote:
> On Thu, Jul 21, 2016 at 2:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I see that the core tests fall over in Turkish still :-(
>
> Turkish has never passed (at least back to 9.0).  It looks like it is
> in the stemming functions.  I don't understand why, I would think
> everything other than English would be failing those if the regression
> tests hard-code English stemming expectations but fail to arrange for
> English stemming rules.

If something fails for Turkish but not other languages it is usually due 
to the upper/lower casing rules of the dotted and the dotless I (I -> ı 
and İ -> i rather than most languages which have I -> i).

Andreas

On Thu, Jul 21, 2016 at 03:53:45PM -0400, Andrew Dunstan wrote:
> 
> 
> On 07/21/2016 02:26 PM, Greg Stark wrote:
> >On Thu, Jul 21, 2016 at 5:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>Confirmed here.  Will deal with it, but I wonder why we have no buildfarm
> >>members covering this ...
> >We're not going to have a build farm member for every locale the local
> >systems support.
> >
> >Perhaps the build farm script should pick a random locale for each
> >run. Either a random locale from the set on the OS or a random
> >language from a list of locale that the regression tests are intended
> >to be safe for.
> >
> 
> 
> I don't see why we shouldn't have a buildfarm machine that tests a very
> large number of locales. It takes a very lightly resourced machine like
> nightjar just over two minutes per locale. The list of locales to test is a
> setting in the config file.

+1.  Ten animals of ~75 locales apiece would give fair per-animal runtime.

On 21/07 08.42, Jeff Janes wrote:
> In Danish, the sequence 'aa' is sometimes treated as a single letter
> which collates after 'z'.

For the record: this is also true for Norwegian, in both locales it
collates equal to 'Ã¥' which is the 29th letter of the alphabet. But
'aa' is no longer used in ordinary words, only names (in Norwegian
only personal names, in Danish also place names).

- Bjorn Munch