Обсуждение: Locale by default?

It occurred to me that a server with locale features that is started in
the C locale is going to behave the same as a server without locale
features.  The exception are a few extra memory moving operations.  (I
sincerely hope that all systems' libcs have optimized paths for the C
locale.)  So we could get rid of this --enable-locale switch altogether.
Given our international user base, this would be an appropriate step and
move the locale support out of the "cumbersome secondary feature"
compartment.  What do you think?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

Hi Peter,

Any idea of how many "extra memory moving operations" that would be?

:-)

Regards and best wishes,

Justin Clift


Peter Eisentraut wrote:
> 
> It occurred to me that a server with locale features that is started in
> the C locale is going to behave the same as a server without locale
> features.  The exception are a few extra memory moving operations.  (I
> sincerely hope that all systems' libcs have optimized paths for the C
> locale.)  So we could get rid of this --enable-locale switch altogether.
> Given our international user base, this would be an appropriate step and
> move the locale support out of the "cumbersome secondary feature"
> compartment.  What do you think?
> 
> --
> Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."    - Indira Gandhi

Peter Eisentraut <peter_e@gmx.net> writes:
> (I sincerely hope that all systems' libcs have optimized paths for the C
> locale.)  So we could get rid of this --enable-locale switch
> altogether.

Some experimental evidence to support the claim that --enable-locale has
zero cost would be good before taking this step.

If any hotspots turn up, we could possibly do runtime checks:
if (locale_is_c())    strcmp()else    strcoll()
        regards, tom lane

If it's of any assistance, I'm working with the Open Source Database
Benchmark guys (osdb.sourceforge.net) to get an AS3AP-based benchmark
for PostgreSQL 7.1.x+ up-and-running reliably.

It's working on my Mandrake Linux 8.0 system here, but I need the main
OSDB guy to get back from holidays to review and commit things to their
CVS.  ETA of around a week from right now.  :)

My point is, if we've got decent benchmarking software (and we can
actually freely use it), we can do real-world validation tests when
considering things like Peter's suggestion.

Sounds good to me.

Regards and best wishes,

Justin Clift


Tom Lane wrote:
> 
> Peter Eisentraut <peter_e@gmx.net> writes:
> > (I sincerely hope that all systems' libcs have optimized paths for the C
> > locale.)  So we could get rid of this --enable-locale switch
> > altogether.
> 
> Some experimental evidence to support the claim that --enable-locale has
> zero cost would be good before taking this step.
> 
> If any hotspots turn up, we could possibly do runtime checks:
> 
>         if (locale_is_c())
>                 strcmp()
>         else
>                 strcoll()
> 
>                         regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."    - Indira Gandhi

> It occurred to me that a server with locale features that is started in
> the C locale is going to behave the same as a server without locale
> features.  The exception are a few extra memory moving operations.  (I
> sincerely hope that all systems' libcs have optimized paths for the C
> locale.)  So we could get rid of this --enable-locale switch altogether.
> Given our international user base, this would be an appropriate step and
> move the locale support out of the "cumbersome secondary feature"
> compartment.  What do you think?

I wouldn't object it if there is a way to disable locale support.  We
in Japan are always troubled by borken Japanese locales on some
systems.  I'm afraid to hear more complains if there is no way to
disable the locale support.  Moreover, collation of locales for
Japanese are broken on all platforms as far as I know. I'm not sure
about other Asian languages though.
--
Tatsuo Ishii

Tatsuo Ishii writes:

> I wouldn't object it if there is a way to disable locale support.

export LC_ALL=C

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

Peter Eisentraut wrote:
> 
> Tatsuo Ishii writes:
> 
> > I wouldn't object it if there is a way to disable locale support.
> 
> export LC_ALL=C

I would object even if there's such a way.
People in Japan have hardly noticed that the strange
behabior is due to the strange locale(LC_COLLATE).

regards,
Hiroshi Inoue

> Tatsuo Ishii writes:
> 
> > I wouldn't object it if there is a way to disable locale support.
> 
> export LC_ALL=C

It's not a solution. My point is people should not be troubled by the
useless feature (at least for Japanese) even if they set their locale
other than C.
--
Tatsuo Ishii

Hiroshi Inoue writes:

> I would object even if there's such a way.
> People in Japan have hardly noticed that the strange
> behabior is due to the strange locale(LC_COLLATE).

I don't think we should design our systems in a way that inconveniences
many users because some users are using broken operating systems.  If
Japanese users have not realized yet that the locale support they are
using is broken, then it's not the right solution to disable it in
PostgreSQL by default.  In that case the problem would just persist for
the system as a whole.  The right solution is for them to turn off locale
support in their operating system, the way it's supposed to be done.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

Tatsuo Ishii writes:

> > Tatsuo Ishii writes:
> >
> > > I wouldn't object it if there is a way to disable locale support.
> >
> > export LC_ALL=C
>
> It's not a solution. My point is people should not be troubled by the
> useless feature (at least for Japanese) even if they set their locale
> other than C.

If people set their locale to something other than C they have evidently
judged that locale is not useless.  Why would they set it otherwise?  I
don't think hiding away a feature because you think it's useless is a good
idea.  If people don't like it, allow them to turn it off.  If there are
potential problems related to the feature, document them.

Face it, everything has locale support these day.  PostgreSQL is one of
the few packages that even has it as an option to turn it off.  Users of
binary packages of PostgreSQL are all invariably faced with locale
features.  So it's not like sudden unasked-for locale support is going to
be a major shock.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

> > I would object even if there's such a way.
> > People in Japan have hardly noticed that the strange
> > behabior is due to the strange locale(LC_COLLATE).
> 
> I don't think we should design our systems in a way that
inconveniences
> many users because some users are using broken operating systems.  If
> Japanese users have not realized yet that the locale support they are
> using is broken, then it's not the right solution to disable it in
> PostgreSQL by default.  In that case the problem would just persist
for
> the system as a whole.  The right solution is for them to turn off
locale
> support in their operating system, the way it's supposed to be done.

I do not agree with your above statement, I would also want a way to
turn 
it off in PostreSQL alone and leave the OS and rest as is (without a
need 
to worry about). (Our admins use C, En_US, De_DE, De_AT here, but no
locale 
support in the db)

Imho we also need to keep in mind that other DB's don't create locale
aware
char columns by default eighter (they have nchar or some other extended 
create table syntax).

Andreas

> Face it, everything has locale support these day.  PostgreSQL is one
of
> the few packages that even has it as an option to turn it off.  Users
of
> binary packages of PostgreSQL are all invariably faced with locale
> features.  So it's not like sudden unasked-for locale support is going
to
> be a major shock.

What makes you so opposed to a GUC for disabling locale support ?

Andreas

Zeugswetter Andreas SB SD writes:

> What makes you so opposed to a GUC for disabling locale support ?

Nothing.  It may in fact be the best solution.

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

Peter Eisentraut <peter_e@gmx.net> writes:
> Zeugswetter Andreas SB SD writes:
>> What makes you so opposed to a GUC for disabling locale support ?

> Nothing.  It may in fact be the best solution.

As long as locale has to be an initdb-time setting, a GUC var won't
help much.
        regards, tom lane

Peter Eisentraut wrote:
> 
> Hiroshi Inoue writes:
> 
> > I would object even if there's such a way.
> > People in Japan have hardly noticed that the strange
> > behabior is due to the strange locale(LC_COLLATE).
> 
> I don't think we should design our systems in a way that inconveniences
> many users because some users are using broken operating systems. If
> Japanese users have not realized yet that the locale support they are
> using is broken,

I don't know if the locale support is broken in Japan.
I can't think of any reasonable Japanese collating sequence
at once(maybe for ever).  I don't think people should know 
about the existence of collating sequences in Japan.

> then it's not the right solution to disable it in
> PostgreSQL by default.  In that case the problem would just persist for
> the system as a whole.  The right solution is for them to turn off locale
> support in their operating system, the way it's supposed to be done.

DBMS should be independent from the OS settings as far as
possible especially in the handling of data. Currently we
could hardly judge if we are running on a locale or not from
the dbms POV and it doesn't seem a dbms kind of thing in the
first place. I'm a dbms guy not an OS guy and really dislike
the requirement for users to export LC_ALL=C. 

regards,
Hiroshi Inoue

> If people set their locale to something other than C they have evidently
> judged that locale is not useless.  Why would they set it otherwise?

As Hiroshi pointed out, the broken thing is the LC_COLLATE, other
things in the local are working.

> I
> don't think hiding away a feature because you think it's useless is a good
> idea.  If people don't like it, allow them to turn it off.  If there are
> potential problems related to the feature, document them.

I don't object the idea letting users turn it off. I said we need a
way to turn it off in the configuration/compile time.

> Face it, everything has locale support these day.  PostgreSQL is one of
> the few packages that even has it as an option to turn it off.  Users of
> binary packages of PostgreSQL are all invariably faced with locale
> features.  So it's not like sudden unasked-for locale support is going to
> be a major shock.

I would say it's a misunderstanding that the locale (more precisely
LC_COLLATE) is usefull for *any* Language/encodings.
--
Tatsuo Ishii

> Hiroshi Inoue writes:
> 
> > I would object even if there's such a way.
> > People in Japan have hardly noticed that the strange
> > behabior is due to the strange locale(LC_COLLATE).
> 
> I don't think we should design our systems in a way that inconveniences
> many users because some users are using broken operating systems.  If

I don't understand why you object the idea giving PostgreSQL the
ability to turn off the locale support in configuration/compile
time. In that way, there's no inconveniences for "many users".
--
Tatsuo Ishii

> DBMS should be independent from the OS settings as far as
> possible especially in the handling of data. Currently we
> could hardly judge if we are running on a locale or not from
> the dbms POV and it doesn't seem a dbms kind of thing in the
> first place. I'm a dbms guy not an OS guy and really dislike
> the requirement for users to export LC_ALL=C. 

Yup, I can second that.
Also note, that currently a locale aware index might get corrupted if 
you do an OS upgrade (that changes the collation: e.g. add the ?
symbol). 
I sortof think, that pg locale support is not yet up to prime time. 
If we had something that conformed to the Spec (per column lang and 
collation), then yes I would make it mainstream, but as is ?

Andreas

Tatsuo Ishii writes:

> I don't understand why you object the idea giving PostgreSQL the
> ability to turn off the locale support in configuration/compile
> time. In that way, there's no inconveniences for "many users".

I don't mind at all the ability to turn it off.  My point is that the
compile time is the wrong time to do it.  Many users use binary
packages these days, many more users would like to use binary packages.
But the creators of these packages have to make configuration choices to
satisfy all of their users.  So they turn on the locale support, because
that way if you don't want it you can turn if off.  The other way around
doesn't work.

The more appropriate way to handle this situation is to make it a runtime
option.  I agree that the LC_ALL/LC_COLLATE/LANG lattice is confusing and
fragile.  But there can be other ways, e.g.,

initdb --locale=en_US
initdb --locale-collate=C --locale-ctype=en_US
initdb # defaults to --locale=C

or in postgresql.conf

locale=C
locale_numeric=en_US
etc.

or

SHOW locale;
SHOW locale_numeric;

That way you always know exactly what situation you're in.  I think this
was Hiroshi's main concern, the reliance on export LC_ALL, and I agree
that this is bad.

You say locale in Japan works, except for LC_COLLATE.  This concern would
be satisfied by the above approach.

Comments?

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter

> > I don't understand why you object the idea giving PostgreSQL the
> > ability to turn off the locale support in configuration/compile
> > time. In that way, there's no inconveniences for "many users".
> 
> I don't mind at all the ability to turn it off.  My point is that the
> compile time is the wrong time to do it.  Many users use binary
> packages these days, many more users would like to use binary
packages.
> But the creators of these packages have to make configuration choices
to
> satisfy all of their users.  So they turn on the locale support,
because
> that way if you don't want it you can turn if off.  The other way
around
> doesn't work.

Yup, imho we all understood that and the only (to be validated) concern
is
performance.

> 
> The more appropriate way to handle this situation is to make it a
runtime
> option.  I agree that the LC_ALL/LC_COLLATE/LANG lattice is confusing
and
> fragile.  But there can be other ways, e.g.,

Yes, that was the (or at least my) main concern.
> initdb --locale=en_US
> initdb --locale-collate=C --locale-ctype=en_US
> initdb # defaults to --locale=C
> 
> or in postgresql.conf
> 
> locale=C
> locale_numeric=en_US
> etc.
> 
> or
> 
> SHOW locale;
> SHOW locale_numeric;
> 
> That way you always know exactly what situation you're in.  I think
this
> was Hiroshi's main concern, the reliance on export LC_ALL, and I agree
> that this is bad.
> 
> You say locale in Japan works, except for LC_COLLATE.  This concern
would
> be satisfied by the above approach.
> 
> Comments?

I think that's it :-)

Andreas

> Face it, everything has locale support these day.  PostgreSQL is one of
> the few packages that even has it as an option to turn it off.  Users of
> binary packages of PostgreSQL are all invariably faced with locale
> features.  So it's not like sudden unasked-for locale support is going to
> be a major shock.

Certainly everyone would agree that "locale support" is desirable.
Tatsuo has been one of the earliest and most vocal participants in
design speculations on how to support the SQL9x concept of character
sets and collations, which for purposes of long range planning seem to
be synonymous with "locale" afaict.

The question is whether and how to continue to extend the use of
OS-supplied features to accomplish this support, with the severe
restrictions (from an SQL9x pov) which come with the OS implementation.
                  - Thomas

> > Face it, everything has locale support these day.  PostgreSQL is one of
> > the few packages that even has it as an option to turn it off.  Users of
> > binary packages of PostgreSQL are all invariably faced with locale
> > features.  So it's not like sudden unasked-for locale support is going to
> > be a major shock.
> 
> Certainly everyone would agree that "locale support" is desirable.

No. At least for Japanese, LC_COLLATE is not usefull at all.  Let me
explain why. Japanese has three kind of characters: The first one is
called "Kanji", Scond one is "Hiragana". The last one is "Katakana".
Many pary of data stored in database are usually written in Kanji. The
problem is, Kanji is an ideogram and there is no algorithm to guess
the correct pronunciation for Kanji letters. The only solution for
this is add a separate column having Hiragana or Katakana letters
which represents the pronunciation for the Kanji column (Hiragana and
Kataka are phonogram). Sorting is also done by the additional
Hiragana/Katakan column, that can be done according to the code point
of Hiragana/Katakana. So no locale support (LC_COLLATE) is neccessary
at all for Japanese.

> Tatsuo has been one of the earliest and most vocal participants in
> design speculations on how to support the SQL9x concept of character
> sets and collations, which for purposes of long range planning seem to
> be synonymous with "locale" afaict.
> 
> The question is whether and how to continue to extend the use of
> OS-supplied features to accomplish this support, with the severe
> restrictions (from an SQL9x pov) which come with the OS implementation.

In my opinion, with the SQL99 collate support, the current locale
support should be vanished.
--
Tatsuo Ishii

> > Certainly everyone would agree that "locale support" is desirable.
> No...

That is why I put "locale support" in double-quotes. Sorry that I was
cryptic, but I do understand your concern that OS-specific locale
support is suspect for some languages.

> In my opinion, with the SQL99 collate support, the current locale
> support should be vanished.

Right. That is my opinion too.
                  - Thomas