Re: Character Encoding Question

On Thu, Mar 28, 2013 at 3:35 PM, Don Parris <parrisdc@gmail.com> wrote:
> Hi all,
>
> I am using Psycopg2 with Python3 & PostgreSQL 9.1 with a database encoded as
> UTF-8 (Kubuntu 12.10).  I have a question about the encoding and decoding,
> and why some fetch calls did what I wanted while fetchall() seemed to choke.
> I got the following error message when calling fetchall():
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9:
> ordinal not in range(128)
>
> The traceback pointed to the fetchall() statement.
>
> First, I was able to do a pattern matching search on a specific field in my
> payee table and could get back whatever records matched the user input in my
> script.  Then I ran a simple SELECT on the same table, using fetchall(), but
> that choked.
>
> I decided to experiment some before asking questions here on the list, and
> discovered that, if I call fetchone(), I got a single record with the fields
> presented as rows in my Bash console.  Then I tried fetchmany() and got the
> same record presented as a tuple.
>
> In the psycopg usage doc
> (http://www.initd.org/psycopg/docs/usage.html#unicode-handling), I found
> this reference:
>
> conn.set_client_encoding('LATIN9')
>
> I tried that, but set the encoding to UTF-8 instead of the LATIN9, supposing
> that might be useful.  Given the successful result, it appears that was just
> what the doctor ordered!

Yes, that should be the case. What was the original encoding? Can you
check "conn.client_encoding" (to get what psycopg thinks the encoding
is) and get the result from the query "show client_encoding" (to get
what the database say it is)? It's strange your connection is not in
utf8 encoding by default.


> My first question is, is this really all I need to do - just set the client
> encoding?  Or is there something else I need to do or be aware of?

Yes, but as said it is suspicious it wasn't already in utf8.


> My second question is why does fetchall() fail without setting the client
> encoding (while the other fetch calls work)?

Are you sure the error is in fetchall itself and not somewhere
downstream? The decoding code path should be common for all the
fetch*() methods. Can you produce a synthetic test case with a couple
of rows in a table to trigger the error?

What psycopg version are you using?

Thanks,

-- Daniele