I think I've decided to only allow escapes of non-ASCII characters
when the database encoding is UTF8. For example, $$"\u2013"$$::json
will fail if the database encoding is WIN1252, even though WIN1252 can
encode U+2013 (EN DASH). This may be somewhat draconian, given that:
* SQL_ASCII can otherwise handle "any" language according to the documentation.
* The XML type doesn't have this restriction (it just stores the
input text verbatim, and converts it to UTF-8 before doing anything
complicated with it).
However, it's simple to implement and understand. The JSON data type
will not perform any automatic conversion between character encodings.Also, if we want to handle this any better in the
future,we won't
have to support legacy data containing a mixture of encodings.
In the future, we could create functions to compensate for the issues
people encounter; for example:
* json_escape_unicode(json [, replace bool]) returns text -- convert
non-ASCII characters to escapes. Optionally, use \uFFFD for
unconvertible characters.* json_unescape_unicode(text [, replace text]) returns json -- like
json_in, but convert Unicode escapes to characters when possible.
Optionally, replace unconvertible characters with a given string.
I've been going back and forth on how to handle encodings in the JSON
type for a while, but suggestions and objections are still welcome.
However, I plan to proceed in this direction so progress can be made.
On another matter, should the JSON type guard against duplicate member
keys? The JSON RFC says "The names within an object SHOULD be
unique," meaning JSON with duplicate members can be considered valid.
JavaScript interpreters (the ones I tried), PHP, and Python all have
the same behavior: discard the first member in favor of the second.
That is, {"key":1,"key":2} becomes {"key":2}. The XML type throws an
error if a duplicate attribute is present (e.g. '<a href="b"
href="c"/>'::xml).
Thanks for the input,
- Joey