Re: Unicode normalization

On Wed, Sep 16, 2009 at 4:42 PM, Sam Mason <sam@samason.me.uk> wrote:
> On Wed, Sep 16, 2009 at 09:35:02PM +0200, Andreas Kalsch wrote:
>> CREATE OR REPLACE FUNCTION test (str text)
>>  RETURNS text
>> AS $$
>>    import unicodedata
>>    return unicodedata.normalize('NFKD', str.decode('UTF-8'))
>> $$ LANGUAGE plpythonu;
>
> I'd guess you want that to be:
>
>  return unicodedata.normalize('NFKD', str.decode('UTF-8')).encode('UTF-8');
>
> If you're converting from a utf8 encoding, you probably need to go
> back again!  This could certainly be made easier though, PG knows what
> encoding its strings are stored in, why doesn't it work with unicode
> strings by default?

Isn't it python that's making the mistake here, not pg?