We don't pass a encoding parameter so libXML2 expecting "UTF8" or expecting correct encoding decl in XML document. When we pass incorrect document - XML is in database encoding, but encoding decl is original, then it should to fail.
the regress test can looks like your (but all chars are valid there)
postgres=# do $$
declare str text;
begin
if current_setting('server_encoding') <> 'UTF8' then return; end if;
This fix should be apply to xmltable function too.
patch attached
It doesn't fix xpath and xmltable functions issues when server encoding is not UTF8. Looks so XPath functions from libXML2 requires UTF8 encoded strings and the result is in UTF8 too - so result should be recoded to server encoding.
I didn't find any info how to enable libXML2 XPath functions for other encoding than UTF8 :( ??