RE: [HACKERS] Lex and things...

>> > Shot, Leon.  The patch removes the #define YY_USES_REJECT from scan.c,
which
>> > means we now have expandable tokens.  Of course, it also removes the
>> > scanning of "embedded minuses", which apparently causes the optimizer
to
>> > unoptimize a little. 
>> 
>> Oh, no. Unary minus gets to grammar parser and there is recognized as
>> such. Then for numeric constants it becomes an *embedded* minus in
>> function doNegate. So unary minus after parser in numeric constants
>> is embedded minus, as it was earlier before patch. In other words,
>> I can see no change in representation of grammar after patching.
Great.
>> 
>> > However, the next step is attacking the limit on the
>> > size of string literals.  These seemed to be wired to YY_BUF_SIZE, or
>> > something.  Is there any reason for this?
>> 
>> Hmm. There is something going on to remove fixed length limits 
>> entirely, maybe someone is already doing something to lexer in
>> that respect? If no, I could look at what can be done there.
Yes, me.  I've removed the query string limit from psql, libpq, and as much
of the backend as I can see.  I have done some (very) preliminary testing,
and managed to get a 95kB query to execute.  However, the two remaining
problems that I have run into so far are token size (which you have just
removed, many thanks ;-), and string literals, which are limited, it seems
to YY_BUF_SIZE (I think).

You see, if I can get the query string limited removed, perhaps someone who
knows a bit more than I do will do something like, hmmm, say, remove the
block size limit from tuple size... hint, hint... anybody...

MikeA


>> 
>> -- 
>> Leon.
>>