Re: [HACKERS] Postgres' lexer

> When I first read that, I thought "can sign every word of that".
> But suddenly realized that there are more buggy situations here:
> consider a>-2. It is parsed as (a) >- (2). Even in original
> Thomas Lockhart's version there is a bug: it parses a>-b as (a) >- (b).

Bugs can be fixed. We don't always need to perform radical surgery.

> So I decided to simply forbid long operators to end with minus. If you
> think that it is right, here is the patch (today is my patch bomb
> day :). It is to be applied *instead* my earlier today's patch.
> Seems that it is the only more or less clean way to deal with
> big operator/unary minus conflict.

That would disallow an existing built-in operator ("?-", the "is
horizontal?" test; of course, that one's my fault too ;).

This is a great conversation, because at the end of it we are going to
have a more solid parser. But I would suggest that we do at least two
things:

1) generate a *complete* list of test cases. I'll include them in the
regression tests to make sure that we preserve capabilities when
changes are made in the future. This should include cases which we
think *should* change behavior later.

2) move slowly on patching the parser for this, since we clearly have
incomplete coverage in our regression tests and since we aren't
perfectly predicting the ramifications yet.

My recollection is that my last patches for the lexer stemmed from
trying to fix unary minus behavior for constants used as arguments to
DDL statements like CREATE SEQUENCE/START, but as I did that I started
seeing other cases which weren't handled correctly. I fixed, to my
understanding of what desirable behavior should be, the cases which
involved numeric constants. imho this same consideration should be
given to other expressions just as you are doing now.

The overall parser behavior should meet some criteria, such as (in
decreasing priority):

o don't produce non-intuitive or unexpected results
o fully expose underlying capabilities of the backend
o try to do the right thing in common cases
o try to do the right thing in unusual cases

I'll make the (perhaps incorrect) claim that the current behavior is
about right for numeric constants (common cases involving various
whitespace possibilities work about right once everything is through
the parser). (The "+-" operator is a good unusual case to focus on,
and we may conclude that it isn't done right at the moment.) Where
things happen in the parser can change. If the current behavior can't
be reconciled with improved behaviors with other non-constant
expressions, then maybe it should be sacrificed, but not until we try
hard to improve it, rather than disallow it...
                   - Thomas

-- 
Thomas Lockhart                lockhart@alumni.caltech.edu
South Pasadena, California