Re: Support UTF-8 files with BOM in COPY FROM

Brar Piening <brar@gmx.de> writes:
> Citing from the Unicode FAQ again:

> Q: Where is a BOM useful?
> A: A BOM is useful at the beginning of files that are typed as text, but 
> for which it is not known whether they are in big or little endian 
> format�it can also serve as a hint indicating that the file is in 
> Unicode, as opposed to in a legacy encoding and furthermore, it act as a 
> signature for the specific encoding form used.

Note that the reference to byte order betrays the implicit context
assumption: that we're talking about UTF16 or UTF32 representation.
A BOM in UTF8 data is useless for its intended purpose of disambiguating
byte order.  It could possibly be useful for telling UTF8 data apart
from non-UTF8 data, except for the inconvenient fact that that byte
sequence is not invalid data in non-UTF8 encodings.

BOM is useless in UTF8, no matter what Microsoft thinks.  Any tool that
relies on it to detect UTF8 data has to have a workaround for overriding
that detection, or it's broken to the point of uselessness.
        regards, tom lane