Robert Haas <robertmhaas@gmail.com> writes:
> On Sat, Apr 24, 2010 at 8:07 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> Sounds useful to me, though as a function like suggested in a later
>> email.
> If tool-builders think this is useful, I have no problem with making
> it available. It should be suitably disclaimed: "We reserve the right
> to rip out the entire flex/yacc-based lexer and parser at any time and
> replace them with a hand-coded system written in Prolog that emits
> tokenization information only in ASN.1-encoded pig latin. If massive
> changes in the way this function works - or its complete disappearance
> - are going to make you grumpy, don't call it."
I'm a bit concerned with the vagueness of the goals here. We started
with a request to dump out node trees, ie, post-parsing representation;
but the example use case of syntax highlighting would find that
representation quite useless. (Example: foo::bar and CAST(foo AS bar)
yield the same parse tree.) A syntax highlighter might get some use
out of the lexer-output token stream, but I'm afraid from the proposed
output that people might be expecting more semantic information than
the lexer can provide. The lexer doesn't, for example, have any clue
that some keywords are commands and others aren't; nor any very clear
understanding about the semantic difference between the tokens '='
and ';'.
Also, if all you want is the lexer, it's not that hard to steal psql's
version and adapt it to your purposes. The lexer doesn't change very
fast, and it's not that big either.
Anyway, it certainly wouldn't be hard for an add-on module to provide a
SRF that calls the lexer (or parser) and returns some sort of tabular
representation of the results. I'm just not sure how useful it'll be
in the real world.
regards, tom lane