Обсуждение: Use of "token" vs "lexeme" in text search documentation

The current documentation seems a bit inconsistent in its use of the
terms "token" and "lexeme".  The majority of the text seems to use
"lexeme" exclusively, which is inconsistent with the fact that the
term "token" is exposed by ts_token_type() and friends.  But there
are a few places that seem to use "lexeme" to mean something returned
by a dictionary.

I was considering trying to adopt these conventions:

* What a parser returns is a "token".

* When a dictionary recognizes a token, what it returns is a "lexeme".

This would make the phrase "normalized lexeme" redundant, since we
don't call it a lexeme at all unless it's been normalized.

Comments?

            regards, tom lane

On Mon, 15 Oct 2007, Tom Lane wrote:

> The current documentation seems a bit inconsistent in its use of the
> terms "token" and "lexeme".  The majority of the text seems to use
> "lexeme" exclusively, which is inconsistent with the fact that the
> term "token" is exposed by ts_token_type() and friends.  But there
> are a few places that seem to use "lexeme" to mean something returned
> by a dictionary.
>
> I was considering trying to adopt these conventions:
>
> * What a parser returns is a "token".
>
> * When a dictionary recognizes a token, what it returns is a "lexeme".
>
> This would make the phrase "normalized lexeme" redundant, since we
> don't call it a lexeme at all unless it's been normalized.
>
> Comments?

Hmm, you say what I always thought. I'd be happy if you stress this in
docs.

>
>             regards, tom lane
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83