Обсуждение: [Fwd: Re: tsearch in core patch]
>> How would this work for initdb with locale C?
>
> I'm worrying about that too.
english '{en_GB, en_US, C}'
I suppose, that locale name always has a dot separator exept C locale ---
which is well known exception
> >> How would this work for initdb with locale C?
> >
> > I'm worrying about that too.
>
> english '{en_GB, en_US, C}'
>
> I suppose, that locale name always has a dot separator exept C locale ---
> which is well known exception
So we would have to?:
japanese '{ja_JP, C}'
How would we know C -> japanese?
Also I'm wondering how we could handle texts including Japanese and
English. It's very common in Japan.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
Tatsuo Ishii wrote:
> japanese '{ja_JP, C}'
>
> How would we know C -> japanese?
>
You can't do that. You can't have different languages (not locales)
mapping to the same 'tsearch language' because the stemmer doesn't know
that a specific word is in english or japanese. So you have two options:
(a) disable stemming (b) leave the language set to 'japanese' and see if
it plays well.
-- Euler Taveira de Oliveira http://www.timbira.com/
> Tatsuo Ishii wrote:
>
> > japanese '{ja_JP, C}'
> >
> > How would we know C -> japanese?
> >
> You can't do that. You can't have different languages (not locales)
> mapping to the same 'tsearch language' because the stemmer doesn't know
> that a specific word is in english or japanese. So you have two options:
> (a) disable stemming (b) leave the language set to 'japanese' and see if
> it plays well.
Ok, probably we need to copy the English stemming rule to the one for
Japanese. I think same thing (commonly used English with local
language) can be applied to Chinese and Korean.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> Ok, probably we need to copy the English stemming rule to the one for
> Japanese.
Pardon my ignorance here, but is the concept of stemming even relevant
to Japanese/Chinese/Korean? What little I know about ideographic
languages suggests it wouldn't work well. And surely the specific rules
in the Snowball project's English stemmer wouldn't work.
> I think same thing (commonly used English with local
> language) can be applied to Chinese and Korean.
Well, it's not hard at all to find chunks of English text that have
embedded bits of French, Spanish, or what-have-you, but that's not an
argument for trying to intermix the stemmers. I doubt that such simple
bits of program could tell the language difference well enough to
determine which stemming rules to apply.
regards, tom lane
> Tatsuo Ishii <ishii@sraoss.co.jp> writes: > > Ok, probably we need to copy the English stemming rule to the one for > > Japanese. > > Pardon my ignorance here, but is the concept of stemming even relevant > to Japanese/Chinese/Korean? What little I know about ideographic > languages suggests it wouldn't work well. And surely the specific rules > in the Snowball project's English stemmer wouldn't work. Your undestanding is correct. English stemmer would not work for Japanese "non English" part. What I meant was the "chunks of English text" in Japanese. > > I think same thing (commonly used English with local > > language) can be applied to Chinese and Korean. > > Well, it's not hard at all to find chunks of English text that have > embedded bits of French, Spanish, or what-have-you, but that's not an > argument for trying to intermix the stemmers. I doubt that such simple > bits of program could tell the language difference well enough to > determine which stemming rules to apply. For Japanese, it will be fairly simple: 7bit ASCII range words must be English (Note that mostly used Japanese encodings such as EUC do not allow to mix with ISO 8859). -- Tatsuo Ishii SRA OSS, Inc. Japan
On 6/25/07, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Well, it's not hard at all to find chunks of English text that have > embedded bits of French, Spanish, or what-have-you, but that's not an > argument for trying to intermix the stemmers. I doubt that such simple > bits of program could tell the language difference well enough to > determine which stemming rules to apply. > While I imagine that is probably true of many, if not most, my project in particular would greatly benefit from the ability to mix stemmers. I work with complex bibliographic data, which has language information embedded within records. This is not limited to the record level either. Individual fields within each bibliographic record can be in different langauges. Especially in countries where making software multi-lingual (such as Canada (en_CA/fr_CA)) is a requirement for use in public institutions, the ability to choose a stemmer and stop-word list at will for any particular record will actually provide the exact behavior needed. The obvious generalization from Canada would be to support any mix of languages supported by tsearch2. I can certainly understand the benefit of making the default configuration a simple locale to language map, but there are definitely uses for searching using different stemmers/stop-lists even within the same corpus/index. So, as a datapoint for the discussion, I would ask that the option of multiple languages per DB locale not be removed if it can be at all avoided. Thanks for listening (and for all the great work on getting tsearch into core! :) ... -- Mike Rylander
"Mike Rylander" <mrylander@gmail.com> writes:
> I can certainly understand the benefit of making the default
> configuration a simple locale to language map, but there are
> definitely uses for searching using different stemmers/stop-lists even
> within the same corpus/index. So, as a datapoint for the discussion,
> I would ask that the option of multiple languages per DB locale not be
> removed if it can be at all avoided.
Nobody is proposing that --- the issue here is just how we set up the
"default" configuration.
regards, tom lane
On 6/25/07, Tom Lane <tgl@sss.pgh.pa.us> wrote: > "Mike Rylander" <mrylander@gmail.com> writes: > > I can certainly understand the benefit of making the default > > configuration a simple locale to language map, but there are > > definitely uses for searching using different stemmers/stop-lists even > > within the same corpus/index. So, as a datapoint for the discussion, > > I would ask that the option of multiple languages per DB locale not be > > removed if it can be at all avoided. > > Nobody is proposing that --- the issue here is just how we set up the > "default" configuration. > Then I misunderstood. Sorry for the noise, folks. -- Mike Rylander
Ishii-san, >>> Ok, probably we need to copy the English stemming rule to the one for >>> Japanese. >> Pardon my ignorance here, but is the concept of stemming even relevant >> to Japanese/Chinese/Korean? What little I know about ideographic >> languages suggests it wouldn't work well. And surely the specific rules >> in the Snowball project's English stemmer wouldn't work. > > Your undestanding is correct. English stemmer would not work for > Japanese "non English" part. That reminds me, don't you guys have your own full text search for Japanese? Planning on merging it with the core code anytime soon? --Josh
> Ishii-san, > > >>> Ok, probably we need to copy the English stemming rule to the one for > >>> Japanese. > >> Pardon my ignorance here, but is the concept of stemming even relevant > >> to Japanese/Chinese/Korean? What little I know about ideographic > >> languages suggests it wouldn't work well. And surely the specific rules > >> in the Snowball project's English stemmer wouldn't work. > > > > Your undestanding is correct. English stemmer would not work for > > Japanese "non English" part. > > That reminds me, don't you guys have your own full text search for > Japanese? Planning on merging it with the core code anytime soon? No. Actually Japanese (non English part) does not need stemming at all. However, since Japanese is an agglutinative language, we have to break continuous Japanese string into space separated "words". For example, we need to break: todayisfine into: today is fine (of course those English are just for non-Japanese spearker's understanding, actually they are Japanese). For this we need good dictionary and software. Fortunately we have several kinds of open source softwares for this pupose. Once I have written a PostgreSQL C function envoking one of these software to do the work and it works great with tsearch2. -- Tatsuo Ishii SRA OSS, Inc. Japan