Re: Status report: getting plpgsql to use the core lexer

Tom Lane <tgl@sss.pgh.pa.us> wrote: 
> One problem that wasn't obvious when I started is that if you are
> trying to use a reentrant lexer, Bison insists on including its
> YYSTYPE union in the call signature of the lexer.  Of course,
> YYSTYPE means different things to the core grammar and plpgsql's
> grammar.  I tried to work around that by having an interface layer
> that would (among other duties) translate as needed.  It turned out
> to be a real PITA, not least because you can't include both
> definitions into the same C file.  The scheme I have has more or
> less failed --- I think I'd need *two* interface layers to make it
> work without unmaintainable kluges.  It would probably be better to
> try to adjust the core lexer's API some more so that it does not
> depend on the core YYSTYPE, but I'm not sure yet how to get Bison to
> play along without injecting an interface layer (and hence wasted
> cycles) into the core grammar/lexer interface.
> 
> Another pretty serious issue is that the current plpgsql lexer
> treats various sorts of qualified names as single tokens.  I had
> thought this could be worked around in the interface layer by doing
> more lookahead.  You can do that, and it mostly works, but it's
> mighty tedious.  The big problem is that "yytext" gets out of step
> --- it will point at the last token the core lexer has processed,
> and there's no good way to back it up after lookahead.  I spent a
> fair amount of time trying to work around that by eliminating uses
> of "yytext" in plpgsql, and mostly succeeded, but there are still
> some left.  (Some of the remaining regression failures are error
> messages that point at the wrong token because they rely on yytext.)
> 
> Now, having name lookup happen at the lexical level is pretty bogus
> anyhow.  The long-term solution here is probably to avoid doing
> lookup in the plpgsql lexer and move it into some sort of callback
> hook in the main parser, as we've discussed before.  I didn't want
> to get into that right away, but I'm now thinking it has to happen
> before not after refactoring the lexer code.  One issue that has to
> be surmounted before that can happen is that plpgsql currently
> throws away all knowledge of syntactic scope after initial
> processing of a function --- the "name stack" is no longer available
> when we want to parse individual SQL commands.  We can probably
> rearrange that design but it's another bit of work I don't have time
> for right now.
All of this sounds pretty familiar to me.  As you may recall, our
framework includes a SQL parser which parses the subset of standard
SQL we feel is portable enough, and generates Java classes to
implement the code in "lowest common denominator" SQL with all
procedural code for triggers and stored procedures handled in Java
(which runs in our middle tier database service).  We use ANTLR, and
initially had a three-phase process: lexer, parser, and tree-walkers
to generate code.  We were doing way too much in the parser phase --
checking for table names, column names, data types, etc.  The syntax
of SQL forced us to do a lot of scanning forward and remembering where
we were (especially to get the FROM clause information so we could
process columns in the result list).
We were able to get to much cleaner code by rewriting the parser to
have a "dumb" phase to get the overall structure into an AST, and then
use a tree-walker phase to do all the lookups and type resolution
after we had the rough structure, writing another AST to walk for code
generation.  Besides making the code cleaner and easier to maintain,
it helped us give better error messages pointing more accurately to
the source of the problem.  I don't know if a similar approach is
feasible in flex/bison, but if it is, refactoring for an extra pass
might be worth the trouble.
-Kevin