Обсуждение: changing text search treatment of puncutation

Text with the '/' character gets treated as a file path, e.g.

select * from to_tsvector('english', 'home/work');

gives only the single token:

'home/work':1

Changing '/' to '-' gives

'home':2 'work':3 'home-work':1

which is much more desirable for this application.

Is there an easy way to change '/' to be treated like '-' ? I've
looked over the documentation several times and could not find
anything. Even just a way to get the two tokens 'home' and 'work'
without the joined form would be helpful.

Thanks,


John DeSoi, Ph.D.

John DeSoi <desoi@pgedit.com> writes:
> Is there an easy way to change '/' to be treated like '-' ? I've
> looked over the documentation several times and could not find
> anything. Even just a way to get the two tokens 'home' and 'work'
> without the joined form would be helpful.

Seems like the simplest solution is just to apply
    regexp_replace(text, '/', '-', 'g')
before letting the text search stuff have the string.  If you're
using a trigger to update a tsvector column, this would be pretty
trivial to do within the trigger.

In general there seem to be a lot of ways that people wish they
could tweak the text search parser, and telling them to write
their own parser isn't a very helpful response for most folk.
I don't have an idea about how to improve the situation, but
it seems like something that should be thought about.

            regards, tom lane

> In general there seem to be a lot of ways that people wish they
> could tweak the text search parser, and telling them to write
> their own parser isn't a very helpful response for most folk.
> I don't have an idea about how to improve the situation, but
> it seems like something that should be thought about.

We (with Oleg) thought hard about it and we don't find a solution yet.
Configurable parser should be:
- fast
- flexible
- not error-prone
- comfortable to use by non-programmer (at least for non-C programmer)

It might be a table-driven state machine (just put TParserStateAction into
table(s) with some caching for first step) , but it's complex to operate and
it's needed to prove correctness of changes in states before its become in use.

--
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

On Wed, 2 Jul 2008, Tom Lane wrote:

> John DeSoi <desoi@pgedit.com> writes:
>> Is there an easy way to change '/' to be treated like '-' ? I've
>> looked over the documentation several times and could not find
>> anything. Even just a way to get the two tokens 'home' and 'work'
>> without the joined form would be helpful.
>
> Seems like the simplest solution is just to apply
>     regexp_replace(text, '/', '-', 'g')
> before letting the text search stuff have the string.  If you're
> using a trigger to update a tsvector column, this would be pretty
> trivial to do within the trigger.
>
> In general there seem to be a lot of ways that people wish they
> could tweak the text search parser, and telling them to write
> their own parser isn't a very helpful response for most folk.
> I don't have an idea about how to improve the situation, but
> it seems like something that should be thought about.

Sure, we thought about this. The most difficult part in user-configurable
parser (we thought about table-driven  finite automata) is the foolproof
design. There are should be algorithms for testing validity of finite
automata, but we don't know any effective way.

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83