Re: Latin vs non-Latin words in text search parsing

Tom Lane wrote:

> ISTM that perhaps a more generally useful definition would be
> 
> lword        Only ASCII letters
> nlword        Entirely letters per iswalpha(), but not lword
> word        Entirely alphanumeric per iswalnum(), but not nlword
>         (hence, includes at least one digit)
> 
> However, I am no linguist and maybe I'm missing something.

I tend to agree with the need to redefine the categories.  I am not sure
I agree with this particular definition though.  I would think that a
"latin word" should include ASCII letters and accented letters, and a
non-latin word would be one that included only non-ASCII chars.

alvherre=# select * from ts_debug('spanish', 'añadido añadió añadidura');Alias |  Description  |   Token   |
Dictionaries |      Lexized token       
 
-------+---------------+-----------+----------------+--------------------------word  | Word          | añadido   |
{spanish_stem}| spanish_stem: {añad}blank | Space symbols |           | {}             | word  | Word          | añadió
  | {spanish_stem} | spanish_stem: {añad}blank | Space symbols |           | {}             | word  | Word          |
añadidura| {spanish_stem} | spanish_stem: {añadidur}
 
(5 lignes)

I would think those would all fit in the "latin word" category.  This
example is more interesting because it shows a word categorized
differently just because the plural loses the accent:

alvherre=# select * from ts_debug('spanish', 'caracteres carácter');Alias |  Description  |   Token    |  Dictionaries
|     Lexized token       
 
-------+---------------+------------+----------------+--------------------------lword | Latin word    | caracteres |
{spanish_stem}| spanish_stem: {caracter}blank | Space symbols |            | {}             | word  | Word          |
carácter  | {spanish_stem} | spanish_stem: {caract}
 
(3 lignes)

I am not sure if there are any western european languages were words can
only be formed with non-ascii chars.  At least in spanish accents tend
to be rare.  However, I would think this is also wrong:

alvherre=# select * from ts_debug('french', 'à');Alias  |  Description   | Token | Dictionaries  |  Lexized token  
--------+----------------+-------+---------------+-----------------nlword | Non-latin word | à     | {french_stem} |
french_stem:{}
 
(1 ligne)

I don't think this is much of a problem, this particular word being
(most likely) a stopword.

So, how about

lword        Entirely letters per iswalpha, with at least one ASCII
nlword        Entirely letters per iswalpha
word        Entirely alphanumeric per iswalnum, but not nlword

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support