Обсуждение: garbage in psql -l
Hi there,
I have problem with CVS HEAD (noticed a week or so ago) -
psql -l show garbage instead of -|+. Looks, like utf-8 symbols used
instead that ascii characters.
List of databases Name Б■┌ Owner Б■┌ Encoding Б■┌ Collation
Б■┌ Ctype Б■┌ Access privileges
Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■╪Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■╪Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■╪Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■
╪Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■╪Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─Б■─
contrib_regressionБ■┌ postgres Б■┌ UTF8 Б■┌ ru_RU.UTF-8 Б■┌ ru_RU.UTF-8 Б■┌ nomao Б■┌ postgres Б■┌
UTF8 Б■┌ ru_RU.UTF-8 Б■┌ ru_RU.UTF-8 Б■┌ postgres Б■┌ postgres Б■┌ UTF8 Б■┌ ru_RU.UTF-8 Б■┌
ru_RU.UTF-8Б■┌ template0 Б■┌ postgres Б■┌ UTF8 Б■┌ ru_RU.UTF-8 Б■┌ ru_RU.UTF-8 Б■┌ =c/postgres
Б∙╥ Б∙╥ Б∙╥ Б∙╥ Б∙▌ postgres=CTc/postgres
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
> I have problem with CVS HEAD (noticed a week or so ago) -
> psql -l show garbage instead of -|+. Looks, like utf-8 symbols used
> instead that ascii characters.
Hm, you only see it for -l and not for all tabular output? That's
a bit strange.
regards, tom lane
On Tue, 24 Nov 2009, Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
>> I have problem with CVS HEAD (noticed a week or so ago) -
>> psql -l show garbage instead of -|+. Looks, like utf-8 symbols used
>> instead that ascii characters.
>
> Hm, you only see it for -l and not for all tabular output? That's
> a bit strange.
yes, I'm surprising myself. Teodor has no problem, but he is under FreeBSD,
while I use slackware linux. Here is ldd output.
pg-head@zen:~/cvs/HEAD/pgsql$ ldd /usr/local/pgsql-head/bin/psql linux-gate.so.1 => (0xffffe000)
libpq.so.5=> /usr/local/pgsql-head/lib/libpq.so.5 (0xb7f33000) libz.so.1 => /usr/lib/libz.so.1 (0xb7ef8000)
libreadline.so.5 => /usr/lib/libreadline.so.5 (0xb7ec8000) libtermcap.so.2 => /lib/libtermcap.so.2 (0xb7ec4000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7e92000) libdl.so.2 => /lib/libdl.so.2 (0xb7e8d000)
libm.so.6=> /lib/libm.so.6 (0xb7e67000) libc.so.6 => /lib/libc.so.6 (0xb7d07000) /lib/ld-linux.so.2
(0xb7f4f000)
>
> regards, tom lane
>
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
> On Tue, 24 Nov 2009, Tom Lane wrote:
>> Hm, you only see it for -l and not for all tabular output? That's
>> a bit strange.
> yes, I'm surprising myself. Teodor has no problem, but he is under FreeBSD,
> while I use slackware linux. Here is ldd output.
What's your locale environment? ("env | grep ^L" would help.)
regards, tom lane
On Tue, 24 Nov 2009, Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
>> On Tue, 24 Nov 2009, Tom Lane wrote:
>>> Hm, you only see it for -l and not for all tabular output? That's
>>> a bit strange.
>
>> yes, I'm surprising myself. Teodor has no problem, but he is under FreeBSD,
>> while I use slackware linux. Here is ldd output.
>
> What's your locale environment? ("env | grep ^L" would help.)
LC_COLLATE=ru_RU.KOI8-R
LANG=C
LC_CTYPE=ru_RU.KOI8-R
I had no problem with this.
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
> On Tue, 24 Nov 2009, Tom Lane wrote:
>> What's your locale environment? ("env | grep ^L" would help.)
> LC_COLLATE=ru_RU.KOI8-R
> LANG=C
> LC_CTYPE=ru_RU.KOI8-R
Hmm, I can duplicate the fact that psql -l uses utf8 characters
(because it connects to the postgres DB which has utf8 encoding)
but for me, ordinary selects within psql use the utf8 characters
too. Do you perhaps have something in ~/.psqlrc to force a different
client encoding?
regards, tom lane
On tis, 2009-11-24 at 21:32 +0300, Oleg Bartunov wrote:
> On Tue, 24 Nov 2009, Tom Lane wrote:
>
> > Oleg Bartunov <oleg@sai.msu.su> writes:
> >> On Tue, 24 Nov 2009, Tom Lane wrote:
> >>> Hm, you only see it for -l and not for all tabular output? That's
> >>> a bit strange.
> >
> >> yes, I'm surprising myself. Teodor has no problem, but he is under FreeBSD,
> >> while I use slackware linux. Here is ldd output.
> >
> > What's your locale environment? ("env | grep ^L" would help.)
>
> LC_COLLATE=ru_RU.KOI8-R
> LANG=C
> LC_CTYPE=ru_RU.KOI8-R
>
> I had no problem with this.
Seems like a mismatch between client encoding and actual locale
environment.
On Tue, 24 Nov 2009, Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
>> On Tue, 24 Nov 2009, Tom Lane wrote:
>>> What's your locale environment? ("env | grep ^L" would help.)
>
>> LC_COLLATE=ru_RU.KOI8-R
>> LANG=C
>> LC_CTYPE=ru_RU.KOI8-R
>
> Hmm, I can duplicate the fact that psql -l uses utf8 characters
> (because it connects to the postgres DB which has utf8 encoding)
> but for me, ordinary selects within psql use the utf8 characters
> too. Do you perhaps have something in ~/.psqlrc to force a different
> client encoding?
>
yes,
set client_encoding to KOI8;
but it never hurts me ! I tried to comment it, but it doesn't helped.
Notice, psql from 8.4 works nice.
> regards, tom lane
>
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Tue, 24 Nov 2009, Peter Eisentraut wrote:
> On tis, 2009-11-24 at 21:32 +0300, Oleg Bartunov wrote:
>> On Tue, 24 Nov 2009, Tom Lane wrote:
>>
>>> Oleg Bartunov <oleg@sai.msu.su> writes:
>>>> On Tue, 24 Nov 2009, Tom Lane wrote:
>>>>> Hm, you only see it for -l and not for all tabular output? That's
>>>>> a bit strange.
>>>
>>>> yes, I'm surprising myself. Teodor has no problem, but he is under FreeBSD,
>>>> while I use slackware linux. Here is ldd output.
>>>
>>> What's your locale environment? ("env | grep ^L" would help.)
>>
>> LC_COLLATE=ru_RU.KOI8-R
>> LANG=C
>> LC_CTYPE=ru_RU.KOI8-R
>>
>> I had no problem with this.
>
> Seems like a mismatch between client encoding and actual locale
> environment.
why 8.4 has no real problem ?
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
> why 8.4 has no real problem ?
Because we never tried to use utf8 table decoration before. This
is collateral damage from Roger Leigh's recent patches.
The problem is evidently that Oleg is depending on ~/.psqlrc to
set client_encoding the way he wants it, but that file does not
get read for a "psql -l" invocation. (Probably not for -c either.)
The locale environment really isn't at issue because we do not look
at it to establish client encoding. Perhaps Oleg should be setting
PGCLIENTENCODING instead of depending on ~/.psqlrc, but I suspect
he's not the only one doing it that way.
There has been some talk of altering the rules for setting psql's
default client_encoding. We could think about that, or we could
back off trying to use linestyle=unicode without an explicit setting.
If we do neither, I suspect we'll be hearing more complaints. I'll
bet there are lots of people who are using database encoding = UTF8
but don't actually have unicode-capable terminal setups. It's never
hurt them before, especially not if they aren't really storing any
non-ASCII data.
regards, tom lane
On tis, 2009-11-24 at 21:55 +0300, Oleg Bartunov wrote: > > Seems like a mismatch between client encoding and actual locale > > environment. > > why 8.4 has no real problem ? Because table formatting with Unicode characters is a new feature. Anyway, that patch to set the client encoding automatically from the locale sounds even more useful now.
Peter Eisentraut <peter_e@gmx.net> writes:
> Anyway, that patch to set the client encoding automatically from the
> locale sounds even more useful now.
I think you're being overoptimistic to assume that that's going to
eliminate the issue. It might patch things for Oleg's particular
configuration; but the real problem IMO is that people are depending
on ~/.psqlrc to set encoding/locale related behavior, and that file
isn't read before executing -l/-c (not to mention -X).
I wonder whether the most prudent solution wouldn't be to prevent
default use of linestyle=unicode if ~/.psqlrc hasn't been read.
regards, tom lane
On Tue, 24 Nov 2009, Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
>> why 8.4 has no real problem ?
>
> Because we never tried to use utf8 table decoration before. This
> is collateral damage from Roger Leigh's recent patches.
>
> The problem is evidently that Oleg is depending on ~/.psqlrc to
> set client_encoding the way he wants it, but that file does not
> get read for a "psql -l" invocation. (Probably not for -c either.)
>
> The locale environment really isn't at issue because we do not look
> at it to establish client encoding. Perhaps Oleg should be setting
> PGCLIENTENCODING instead of depending on ~/.psqlrc, but I suspect
> he's not the only one doing it that way.
yes, PGCLIENTENCODING=KOI8 psql -l works as it should be
>
> There has been some talk of altering the rules for setting psql's
> default client_encoding. We could think about that, or we could
> back off trying to use linestyle=unicode without an explicit setting.
> If we do neither, I suspect we'll be hearing more complaints. I'll
> bet there are lots of people who are using database encoding = UTF8
> but don't actually have unicode-capable terminal setups. It's never
> hurt them before, especially not if they aren't really storing any
> non-ASCII data.
what's benefit of using linestyle=unicode ? I like old ASCII style
for console.
>
> regards, tom lane
>
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
> what's benefit of using linestyle=unicode ? I like old ASCII style
> for console.
Well, I have to grant that it looks pretty spiffy on a unicode-enabled
display. Whether that's enough reason to risk breaking things for
people with non-unicode-enabled displays is certainly worth debating.
Maybe we should just make the default be linestyle=ascii all the time,
and tell people to turn it on in their ~/.psqlrc if they want it.
regards, tom lane
On Tue, 24 Nov 2009, Tom Lane wrote:
> Oleg Bartunov <oleg@sai.msu.su> writes:
>> what's benefit of using linestyle=unicode ? I like old ASCII style
>> for console.
>
> Well, I have to grant that it looks pretty spiffy on a unicode-enabled
> display. Whether that's enough reason to risk breaking things for
> people with non-unicode-enabled displays is certainly worth debating.
>
> Maybe we should just make the default be linestyle=ascii all the time,
> and tell people to turn it on in their ~/.psqlrc if they want it.
+1
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Tue, Nov 24, 2009 at 4:49 PM, Oleg Bartunov <oleg@sai.msu.su> wrote: > On Tue, 24 Nov 2009, Tom Lane wrote: > >> Oleg Bartunov <oleg@sai.msu.su> writes: >>> >>> what's benefit of using linestyle=unicode ? I like old ASCII style >>> for console. >> >> Well, I have to grant that it looks pretty spiffy on a unicode-enabled >> display. Whether that's enough reason to risk breaking things for >> people with non-unicode-enabled displays is certainly worth debating. >> >> Maybe we should just make the default be linestyle=ascii all the time, >> and tell people to turn it on in their ~/.psqlrc if they want it. > > +1 +1. ...Robert
On Tue, Nov 24, 2009 at 02:19:27PM -0500, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > Anyway, that patch to set the client encoding automatically from the > > locale sounds even more useful now. > > I think you're being overoptimistic to assume that that's going to > eliminate the issue. It might patch things for Oleg's particular > configuration; but the real problem IMO is that people are depending > on ~/.psqlrc to set encoding/locale related behavior, and that file > isn't read before executing -l/-c (not to mention -X). > > I wonder whether the most prudent solution wouldn't be to prevent > default use of linestyle=unicode if ~/.psqlrc hasn't been read. This problem is caused when there's a mismatch between the client encoding and the user's locale. We can detect this at runtime and fall back to ASCII if we know they are incompatible. Why don't we combine the two approaches we looked at so far: 1) The PG client encoding is UTF-8 2) The user's locale codeset (from nl_langinfo(CODESET)) is UTF-8 If *both* the conditions are satisfied simultaneously then we are guaranteed that things will display correctly given what the user has told us they wanted. If only one is satisfied then we remain using ASCII and problems such as the non-UTF-8-locale mis-display seen here are avoided, while still allowing Unicode display for users who have a UTF-8 locale as well as a UTF-8 client encoding (such as myself ;-) This should be a one-liner patch to update the existing check. Regards, Roger -- .''`. Roger Leigh: :' : Debian GNU/Linux http://people.debian.org/~rleigh/`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
Roger Leigh <rleigh@codelibre.net> writes:
> On Tue, Nov 24, 2009 at 02:19:27PM -0500, Tom Lane wrote:
>> I wonder whether the most prudent solution wouldn't be to prevent
>> default use of linestyle=unicode if ~/.psqlrc hasn't been read.
> This problem is caused when there's a mismatch between the
> client encoding and the user's locale. We can detect this at
> runtime and fall back to ASCII if we know they are incompatible.
Well, no, that is *one* of the possible failure modes. I've hit others
already in the short time that the patch has been installed. The one
that's bit me most is that the locale environment seen by psql doesn't
necessarily match what my xterm at the other end of an ssh connection
is prepared to do --- which is something that psql simply doesn't have
a way to detect. Again, this is something that's never mattered before
unless one was really pushing non-ASCII data around, and even then it
was often possible to be sloppy.
I'd be more excited about finding a way to use linestyle=unicode by
default if it had anything beyond cosmetic benefits. But it doesn't,
and it's hard to justify ratcheting up the requirements for users to get
their configurations exactly straight when that's all they'll get for it.
regards, tom lane
On Tue, Nov 24, 2009 at 05:43:00PM -0500, Tom Lane wrote: > Roger Leigh <rleigh@codelibre.net> writes: > > On Tue, Nov 24, 2009 at 02:19:27PM -0500, Tom Lane wrote: > >> I wonder whether the most prudent solution wouldn't be to prevent > >> default use of linestyle=unicode if ~/.psqlrc hasn't been read. > > > This problem is caused when there's a mismatch between the > > client encoding and the user's locale. We can detect this at > > runtime and fall back to ASCII if we know they are incompatible. > > Well, no, that is *one* of the possible failure modes. I've hit others > already in the short time that the patch has been installed. The one > that's bit me most is that the locale environment seen by psql doesn't > necessarily match what my xterm at the other end of an ssh connection > is prepared to do --- which is something that psql simply doesn't have > a way to detect. Again, this is something that's never mattered before > unless one was really pushing non-ASCII data around, and even then it > was often possible to be sloppy. Sure, but this type of misconfiguration is entirely outside the purview of psql. Everything else on the system, from man(1) to gcc emacs and vi will be sending UTF-8 codes to your terminal for any non-ASCII character they display. While psql using UTF-8 for its tables is certainly exposing the problem, in reality it was already broken, and it's not psql's "fault" for using functionality the system said was available. It would equally break if you stored non-ASCII characters in your UTF-8-encoded database and then ran a SELECT query, since UTF-8 codes would again be sent to the terminal. For the specific case here, where the locale is KOI8-R, we can determine at runtime that this isn't a UTF-8 locale and stay using ASCII. I'll be happy to send a patch in to correct this specific case. At least on GNU/Linux, checking nl_langinfo(CODESET) is considered definitive for testing which character set is available, and it's the standard SUS/POSIX interface for querying the locale. > I'd be more excited about finding a way to use linestyle=unicode by > default if it had anything beyond cosmetic benefits. But it doesn't, > and it's hard to justify ratcheting up the requirements for users to get > their configurations exactly straight when that's all they'll get for it. Bar the lack of nl_langinfo checking, once this is added we will go out of our way to make sure that the system is capable of handling UTF-8. This is, IMHO, the limit of how far i/any/ tool should go to handle things. Worrying about misconfigured terminals, something which is entirely the user's responsiblility, is I think a step too far--going down this road means you'll be artificially limited to ASCII, and the whole point of using nl_langinfo is to allow sensible autoconfiguation, which almost all programs do nowadays. I don't think it makes sense to "penalise" the majority of users with correctly-configured systems because a small minority have a misconfigured terminal input encoding. It is 2009, and all contemporary systems support Unicode, and for the majority it is the default. Every one of the GNU utilities, plus most other free software, localises itself using gettext, which in a UTF-8 locale, even English locales, will transparently recode its output into the locale codeset. This hasn't resulted in major problems for people using these tools; it's been like this way for years now. Regards, Roger -- .''`. Roger Leigh: :' : Debian GNU/Linux http://people.debian.org/~rleigh/`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
On tis, 2009-11-24 at 14:19 -0500, Tom Lane wrote: > I wonder whether the most prudent solution wouldn't be to prevent > default use of linestyle=unicode if ~/.psqlrc hasn't been read. More generally, it would probably be safer if we used linestyle=unicode only if the client encoding has been set on the client by some explicit action, that is, either via PGCLIENTENCODING or an \encoding statement, but *not* when it is just defaulted from the server encoding. Can we easily detect this difference?
On ons, 2009-11-25 at 00:14 +0000, Roger Leigh wrote: > For the specific case here, where the locale is KOI8-R, we can > determine at runtime that this isn't a UTF-8 locale and stay > using ASCII. I'll be happy to send a patch in to correct this > specific case. There is already a proposed patch to set the client encoding from the locale, but it is pending on some other API refactoring. The problem at hand, however, is that psql -l happens before the client encoding settings are applied, which should be fixed in any case.
On tis, 2009-11-24 at 14:19 -0500, Tom Lane wrote: > I think you're being overoptimistic to assume that that's going to > eliminate the issue. It might patch things for Oleg's particular > configuration; but the real problem IMO is that people are depending > on ~/.psqlrc to set encoding/locale related behavior, and that file > isn't read before executing -l/-c (not to mention -X). The -l/-c case should probably be fixed. If the output contains non-ASCII data, then it's not going to display correctly. Not so much a problem for -l, but definitely for -c, and of course with the Unicode line drawing now in fact also for -l.
On Tue, Nov 24, 2009 at 22:35, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Oleg Bartunov <oleg@sai.msu.su> writes: >> what's benefit of using linestyle=unicode ? I like old ASCII style >> for console. > > Well, I have to grant that it looks pretty spiffy on a unicode-enabled > display. Whether that's enough reason to risk breaking things for > people with non-unicode-enabled displays is certainly worth debating. > > Maybe we should just make the default be linestyle=ascii all the time, > and tell people to turn it on in their ~/.psqlrc if they want it. FWIW, this patch also *completely* breaks the default windows installs, which will have the database in UTF-8 but there is *never* any UTF-8 support on the console (only UTF-16). So +1 for not making it the default... Or at least have logic based on the *client* to figure out what the default should be, in case that can be made consistent. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Peter Eisentraut <peter_e@gmx.net> writes:
> On tis, 2009-11-24 at 14:19 -0500, Tom Lane wrote:
>> I think you're being overoptimistic to assume that that's going to
>> eliminate the issue. It might patch things for Oleg's particular
>> configuration; but the real problem IMO is that people are depending
>> on ~/.psqlrc to set encoding/locale related behavior, and that file
>> isn't read before executing -l/-c (not to mention -X).
> The -l/-c case should probably be fixed. If the output contains
> non-ASCII data, then it's not going to display correctly. Not so much a
> problem for -l, but definitely for -c, and of course with the Unicode
> line drawing now in fact also for -l.
I'm not sure that the "fix" won't be worse than the disease here.
The historical behavior is that .psqlrc isn't read before executing
-c commands, and I don't find it difficult at all to imagine that
changing that will break some people's scripts.
regards, tom lane
Magnus Hagander <magnus@hagander.net> writes:
> FWIW, this patch also *completely* breaks the default windows
> installs, which will have the database in UTF-8 but there is *never*
> any UTF-8 support on the console (only UTF-16).
Hmm ... so how do we work with utf-8 data in the database? That
should be the same case AFAICS.
regards, tom lane
On Wed, Nov 25, 2009 at 15:41, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Magnus Hagander <magnus@hagander.net> writes: >> FWIW, this patch also *completely* breaks the default windows >> installs, which will have the database in UTF-8 but there is *never* >> any UTF-8 support on the console (only UTF-16). > > Hmm ... so how do we work with utf-8 data in the database? That > should be the same case AFAICS. It breaks, yes :-), and you have to set the client encoding manually. But it works in the simple case where your data is ASCII for example. Which usually means it's fine for listing databases, or database objects, or such things. But with the unicode line patch, it's basically completely useless, not just partially annoying. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
> On Wed, Nov 25, 2009 at 15:41, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Magnus Hagander <magnus@hagander.net> writes:
>>> FWIW, this patch also *completely* breaks the default windows
>>> installs, which will have the database in UTF-8 but there is *never*
>>> any UTF-8 support on the console (only UTF-16).
>>
>> Hmm ... so how do we work with utf-8 data in the database? �That
>> should be the same case AFAICS.
> It breaks, yes :-), and you have to set the client encoding manually.
> But it works in the simple case where your data is ASCII for example.
> Which usually means it's fine for listing databases, or database
> objects, or such things. But with the unicode line patch, it's
> basically completely useless, not just partially annoying.
Right, so that's the Windows version of the comment I made that people
with ASCII-only data have not previously had to be very careful about
getting their locale/encoding environment lined up just so.
I'm now pretty well convinced that this patch is *not* worth the amount
of pain it will cause if it's invoked by default, no matter what
restrictions we try to place on that. We should just make the default
be linestyle=ascii, period.
A logically-separate issue that bears on this is whether to read .psqlrc
before doing -l/-c commands. I'm still concerned about that, but I
suppose we could tell people to add -X if it breaks things for them.
regards, tom lane
On Wed, November 25, 2009 10:04 am, Tom Lane wrote: > I'm now pretty well convinced that this patch is *not* worth the amount > of pain it will cause if it's invoked by default, no matter what > restrictions we try to place on that. We should just make the default > be linestyle=ascii, period. > +1 I've thought this all along, to be honest. cheers andrew
Andrew Dunstan wrote: > On Wed, November 25, 2009 10:04 am, Tom Lane wrote: > >> I'm now pretty well convinced that this patch is *not* worth the amount >> of pain it will cause if it's invoked by default, no matter what >> restrictions we try to place on that. We should just make the default >> be linestyle=ascii, period. >> > > +1 > > I've thought this all along, to be honest. +1 from me too - I really prefer to be on the more conservative side and keeping ASCII as the default output seems to be the mroe sensible thing for the time being. Stefan
On Wed, Nov 25, 2009 at 09:39:32AM -0500, Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > On tis, 2009-11-24 at 14:19 -0500, Tom Lane wrote:
> >> I think you're being overoptimistic to assume that that's going to
> >> eliminate the issue. It might patch things for Oleg's particular
> >> configuration; but the real problem IMO is that people are depending
> >> on ~/.psqlrc to set encoding/locale related behavior, and that file
> >> isn't read before executing -l/-c (not to mention -X).
>
> > The -l/-c case should probably be fixed. If the output contains
> > non-ASCII data, then it's not going to display correctly. Not so much a
> > problem for -l, but definitely for -c, and of course with the Unicode
> > line drawing now in fact also for -l.
>
> I'm not sure that the "fix" won't be worse than the disease here.
> The historical behavior is that .psqlrc isn't read before executing
> -c commands, and I don't find it difficult at all to imagine that
> changing that will break some people's scripts.
The following patch adds in an nl_langinfo(CODESET) check in
addition to the existing client encoding check. With the
patch applied, unicode table formatting will be enabled only if
these three conditions are met:
1) The system supports nl_langinfo (i.e. is POSIX/SUS) 2) nl_langinfo reports UTF-8 is the locale codeset 3) The
clientencoding is UTF-8
This will fix the issue that was reported (in a KOI8-R locale,
the nl_langinfo check will fail). It will additionally
fix the problem for Microsoft Windows users; since their
systems don't support nl_langinfo, automatic unicode
support will not be compiled in and so the default will always
be ASCII.
With the patch applied, the only group of users which might
be negatively affected are users with a misconfigured terminal
who additionally meet the above three criteria. However, the
two primary groups of users for whom this was a genuine bug
(non-UTF-8 locale and Windows) would find this patch should
ensure psql will continue default to ASCII output. This will
fix the -l/-c problem for them as well.
Regards,
Roger
diff --git a/src/bin/psql/print.c b/src/bin/psql/print.c
index 69d5814..8916ffa 100644
--- a/src/bin/psql/print.c
+++ b/src/bin/psql/print.c
@@ -21,6 +21,9 @@#endif#include <locale.h>
+#ifdef HAVE_LANGINFO_H
+#include <langinfo.h>
+#endif#include "catalog/pg_type.h"#include "pqsignal.h"
@@ -2552,8 +2555,16 @@ get_line_style(const printTableOpt *opt){ if (opt->line_style != NULL) return
opt->line_style;
- else if (opt->encoding == pg_get_utf8_id())
+#if (defined(HAVE_LANGINFO_H) && defined(CODESET))
+ /* If a UTF-8 locale is available, and the client encoding is
+ * also UTF-8, switch to UTF-8 box drawing characters
+ */
+ else if ((pg_strcasecmp(nl_langinfo(CODESET), "UTF-8") == 0 ||
+ pg_strcasecmp(nl_langinfo(CODESET), "utf8") == 0 ||
+ pg_strcasecmp(nl_langinfo(CODESET), "CP65001") == 0) &&
+ opt->encoding == pg_get_utf8_id()) return &pg_utf8format;
+#endif else return &pg_asciiformat;}
-- .''`. Roger Leigh: :' : Debian GNU/Linux http://people.debian.org/~rleigh/`. `' Printing on
GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
Roger Leigh <rleigh@codelibre.net> writes:
> The following patch adds in an nl_langinfo(CODESET) check in
> addition to the existing client encoding check.
I think the consensus is pretty clear that we should just set the
default linestyle to ascii and not try to be cute with it. (I had
already committed a patch to that effect before I saw your message.)
The right way for people who want to see the nice graphics is to put
"\pset linestyle unicode" into their .psqlrc. At this point I think
the only open issue is whether to try to fix things so that .psqlrc
will be executed for non-interactive cases like -c/-l. (Hm, is
.psqlrc executed for "psql -f file"? Should it be?)
regards, tom lane
On Wed, Nov 25, 2009 at 11:34:35PM -0500, Tom Lane wrote: > Roger Leigh <rleigh@codelibre.net> writes: > > The following patch adds in an nl_langinfo(CODESET) check in > > addition to the existing client encoding check. > > I think the consensus is pretty clear that we should just set the > default linestyle to ascii and not try to be cute with it. (I had > already committed a patch to that effect before I saw your message.) OK. It would be nice if this could be revisited at the point the work Peter mentioned about automatically selecting the client encoding based upon the user's locale is ready for committing. > The right way for people who want to see the nice graphics is to put > "\pset linestyle unicode" into their .psqlrc. OK. Regards, Roger -- .''`. Roger Leigh: :' : Debian GNU/Linux http://people.debian.org/~rleigh/`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.