Re: making tsearch2 dictionaries

On Tue, 17 Feb 2004, Oleg Bartunov wrote:

> it's unpredictable  and I still don't get your idea of pipilining, but
> in general, I have nothing agains it.

Oh, well, the idea is that instead of the dictionary searching stopping at
the first dictionary in the chain that returns a lexeme, it would take
each of the lexemes returned and pass them on to the next dictionary in
the chain.

So if I specified numbers were to be handled by my num2english dictionary,
followed by en_stem, and then tried to deal get a vector for "100",
num2english would return "one" and "hundred". Then both "one" and
"hundred" would each be looked up in en_stem, and the union of these
lexems would be the final result.

Similarly, if a latin word gets piped through an ispell dictionary before
being sent to en_stem, each possible spelling would be stemmed.

> Aha, the same way as we handle complex words with hyphen - we return
> the whole word and its parts. So you need to introduce new type of token
> in parser and use synonym dictionary which in one's turn will returns
> the symbol token and human readable word.

Okay, that makes sense. I'll look more into how hyphenated words are being
handled now.