RE: [HACKERS] Postgres' lexer

>> There is the possibility for ambiguity. But it is our responsibility
>> to minimize that ambiguity and to make a predictable system, in the
>> presence of Postgres' unique and valuable features such as type
>> extension. imho this is more important than, for example, allowing
>> infinite-length queries.
I agree that predictability is more important than the limit on the query
length, but I think that they can coexist.  I'm not aware of what the unary
minus recognition in the scanner is about, and why it's important, but if it
is important, then perhaps we can look at implementing it in such a way that
no vltc is created.  This should be possible.  After conversation with Vern
(Paxson, author of flex), it appears that we can, under normal conditions,
use a start condition to allow the same token to be identified.  This
removes the vltc, which in turn, means that we don't limit the length of the
token.  Also, vltcs are major performance degraders.

BTW, Thomas, it's not the query length that is limited by this unary minus
issue, but the token length.  The reason I see this as important is because
it means that, once row size is independent of block size, people will try
to insert large text fields.  A large text field is a single token.
So, at the moment, it's not really an issue, but I was hoping to get it out
the way before the row size issue was tackled, so that when that was
complete, everything just worked ;-)

>> Sorry, what is the performance penalty for that feature, and 
>> how do we measure that against breakage of expected, predictable
behavior?
>> Please quantify.
>> 
>> So far, I'm not a fan of the proposed change; we're giving up behavior
>> that impacts Postgres' unique type extension features for an
>> arbitrarily large query buffer (as opposed to a generously large query
>> buffer, which can be accomplished just by changing the fixed size).
Like I say, I think we can do both (and remove the performance penalty of
the vltc), if we do it right.  Thomas, can you send me enough info about the
unary minus token (basic explanation, gotchas, anything else I should know),
and I'll have a look at using a start condition to implement it.

MikeA