Re: UTF8 conversion differences from v8.1.3 to v8.1.4

Tom Lane wrote:
> Eric Faulhaber <ecf@goldencode.com> writes:
>> Can anyone help me understand why converting the NULL code point (0000)
>> from UTF8 to ISO8859_1 is no longer legal in v8.1.4?
>
> Embedded nulls in text strings have never behaved sanely in PG ... or
> hadn't you noticed?  You'd have been better off passing an empty string,
> because that was effectively what you were getting.
>
>             regards, tom lane

OK, but this particular issue is something quite new to the latest
version.  From utf8_and_iso8859_1.c (utf8_to_iso8859_1 function):

    ...

    unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2);
    unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3);
    int      len = PG_GETARG_INT32(4);
    unsigned short c,
                   c1;

    ...

    while (len > 0)
    {
    c = *src;
        if (c == 0)
            report_invalid_encoding(PG_UTF8, (const char *) src, len);

    ...

This is new code in 8.1.4;  the 8.1.3 version did not screen explicitly
for null bytes.

This has some troubling implications for our runtime layer.  Since the
null byte represents a valid code point in both the database's encoding
(in this case LATIN1) and in the client's encoding (UNICODE/UTF8), I
cannot simply strip out null bytes before handing strings to PG;  they
may well have special meaning to application developers.

Converting varchar/text columns into bytea because they may require
embedded nulls is not an option either, since these are valid strings
and need to be treated as such in our runtime.

Am I stuck at 8.1.3 for the time being?  I'd be happy to create a patch
to resolve this for a future version, but if it is not considered a
defect, it doesn't make sense for me to do that.

Thanks,
Eric