On my way to do something else entirely, I came across a couple of
things that are not very nice about psql's lexer:
1. Somebody broke the no-backtracking property back in 9.0 while adding
quoted variable substitution. According to the flex manual, use of
backtracking creates a performance penalty. We once measured the
backend's lexer as being about a third faster with backtrack avoidance,
and presumably it's about the same for psql's. This is not hard to fix,
but should I consider it a bug fix and back-patch? We've not had
complaints about psql getting slower as of 9.0.
2. The lexer rules associated with variable substitution think that
variable names can consist only of ASCII letters and digits (and
underscores). The psql manual is noncommittal about whether non-ASCII
characters are allowed, but a reasonable person would think that the
rules ought to be the same as the backend's idea of what an identifier
is. Does anybody have a problem with improving that? (I'm not
proposing this part as a bug fix, because it does look a little bit
more invasive to fix, because of the way psql deals with unsafe
multibyte encodings.)
regards, tom lane