Обсуждение: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8

See the related bug report https://www.postgresql.org/message-id/CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com

Currently PostgreSQL accepts structurally well-formed EUC_CN byte sequences such as 0xA2A3 into text columns. The value round-trips when client_encoding is EUC_CN, but fails when client_encoding is UTF8 because euc_cn_to_utf8 has no mapping.

If this behavior is intentional for compatibility, the documentation should explicitly say that validation for some legacy encodings is byte-structure validation, not mapping-table validation.
If it is not intentional, stricter validation could reject unassigned byte positions at input time.

--

Zhongpu Chen

On Friday, May 1, 2026, Zhongpu Chen <chenloveit@gmail.com> wrote:

The issue is not specific to E'\\x..' literals. A normal COPY FROM data file with ENCODING 'EUC_CN' can create text rows that later cannot be retrieved with SELECT.

 This suggests that input validation for EUC_CN is only structural, while the EUC_CN-to-UTF8 conversion table is stricter.

I suspect a lack of desire to maintain and ensure that specific values are verified; or accepting the runtime cost to do so.  It is indeed structural.  This point should probably be documented better.  But it’s hard to feel too bad if the input claims it is providing verifiable EUC_CN data then proceeds to supply data that lacks meaning in reality.  We are happy to just store and return your data to you - but it’s unreasonable to ask for it to be converted.  It would be nice for the database to provide an extra layer of protection, so I’m not against the idea.  Either automatically or or at least providing a function that could, say, be called in a trigger for opt-in.  But definitely feels like a problematic benefit-to-cost proposition.

David J.

Thanks for the clarification.

I agree that validation on every input may have runtime-cost concerns. But this can be well-controlled. For example, MySQL adopts a finer checking for EUC-CN (i.e., GB2312) in https://github.com/mysql/mysql-server/blob/trunk/strings/ctype-gb2312.cc:

```

static int func_gb2312_uni_onechar(int code) {
  if ((code >= 0x2121) && (code <= 0x2658))
    return (tab_gb2312_uni0[code - 0x2121]);
  if ((code >= 0x2721) && (code <= 0x296F))
    return (tab_gb2312_uni1[code - 0x2721]);
  if ((code >= 0x3021) && (code <= 0x777E))
    return (tab_gb2312_uni2[code - 0x3021]);
  return (0);
}

```

where `code` is obtained by subtracting 0x8080. Of course, MySQL's checking can also be enhanced.

Anyway, it is reasonable to note these details in the documentation.

On Sat, May 2, 2026 at 11:28 AM David G. Johnston <david.g.johnston@gmail.com> wrote:

On Friday, May 1, 2026, Zhongpu Chen <chenloveit@gmail.com> wrote:

The issue is not specific to E'\\x..' literals. A normal COPY FROM data file with ENCODING 'EUC_CN' can create text rows that later cannot be retrieved with SELECT.

 This suggests that input validation for EUC_CN is only structural, while the EUC_CN-to-UTF8 conversion table is stricter.

I suspect a lack of desire to maintain and ensure that specific values are verified; or accepting the runtime cost to do so.  It is indeed structural.  This point should probably be documented better.  But it’s hard to feel too bad if the input claims it is providing verifiable EUC_CN data then proceeds to supply data that lacks meaning in reality.  We are happy to just store and return your data to you - but it’s unreasonable to ask for it to be converted.  It would be nice for the database to provide an extra layer of protection, so I’m not against the idea.  Either automatically or or at least providing a function that could, say, be called in a trigger for opt-in.  But definitely feels like a problematic benefit-to-cost proposition.

David J.

--

Zhongpu Chen

On 02.05.26 04:31, Zhongpu Chen wrote:
> See the related bug report https://www.postgresql.org/message-id/
> CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com
> <https://www.postgresql.org/message-id/
> CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com>
>
> Currently PostgreSQL accepts structurally well-formed EUC_CN byte
> sequences such as 0xA2A3 into text columns. The value round-trips when
> client_encoding is EUC_CN, but fails when client_encoding is UTF8
> because euc_cn_to_utf8 has no mapping.
>
> If this behavior is intentional for compatibility, the documentation
> should explicitly say that validation for some legacy encodings is byte-
> structure validation, not mapping-table validation.
> If it is not intentional, stricter validation could reject unassigned
> byte positions at input time.

It is in general not necessarily required that all text in all non-UTF8
encodings must be convertible to UTF8.

(This is also a result of history: These encodings were implemented in
PostgreSQL before Unicode.)

That said, I can see how different behaviors might be desirable.

My first question would be, are these non-convertible byte sequences
just characters that don't map to Unicode, or are they invalid within
the definition of the EUC-* encodings themselves?  If the latter, then
we should just reject them (modulo some backward compatibility), similar
to how we reject certain Unicode code points that exist "structurally"
but are not valid for one reason or another.

Alternatively, if these byte sequences are valid characters but they
just didn't end up in Unicode for some reason, then rejecting them might
break valid uses.

(I don't know much about EUC-* to be able to answer these.)