Обсуждение: TSearch2 memory usage

We have a fixed amount of 16 PG backend processes. Once created they
stay forever. OLTP load is distributed over them in a SQF fashion.

As loading a TSearch dictionary takes a few moments, we have a script
that connects to each backend on start up and pushes the dictionary into
ram by simply calling ts_debug('foo');

The dictionary has a file size of 9.8MB.

If we don't call the script `free' prints:

              total       used       free     shared    buffers     cached
Mem:       4048056     953192    3094864          0          4     359300
-/+ buffers/cache:     593888    3454168

After the script has called ts_debug('foo') on each backend:

              total       used       free     shared    buffers     cached
Mem:       4048056    2374508    1673548          0          4     370340
-/+ buffers/cache:    2004164    2043892


Is it supposed to blast so much memory?

This is PG 8.2.4 on x86_64.


--
Regards,
Hannes Dorbath


--
Regards,
Hannes Dorbath

On Mon, 21 May 2007, Hannes Dorbath wrote:

> We have a fixed amount of 16 PG backend processes. Once created they stay
> forever. OLTP load is distributed over them in a SQF fashion.
>
> As loading a TSearch dictionary takes a few moments, we have a script that
> connects to each backend on start up and pushes the dictionary into ram by
> simply calling ts_debug('foo');
>
> The dictionary has a file size of 9.8MB.
>
> If we don't call the script `free' prints:
>
>             total       used       free     shared    buffers     cached
> Mem:       4048056     953192    3094864          0          4     359300
> -/+ buffers/cache:     593888    3454168
>
> After the script has called ts_debug('foo') on each backend:
>
>             total       used       free     shared    buffers     cached
> Mem:       4048056    2374508    1673548          0          4     370340
> -/+ buffers/cache:    2004164    2043892
>
>
> Is it supposed to blast so much memory?

dictionary is just a program ! Are you usin custom dictionaries ?

what does 'select * from ts_debug('foo');' prints ?

>
> This is PG 8.2.4 on x86_64.
>
>
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On 21.05.2007 11:42, Oleg Bartunov wrote:
> dictionary is just a program ! Are you usin custom dictionaries ?

Depends on what you mean, it's a standard ispell dict, defined as:

DictFile="/foo/bar/german_utf8.dict",AffFile="/foo/bar/german_utf8.aff"

You can grab both files here: http://falcon.imos.net/dict.bz2

IIRC I have created them from an older, german OO dict, iconv'd them to
UTF-8 and run compound.pl on them.

The compound stuff stopped to function on 8.2 and as I couldn't find a
solution I stripped the compound z line from the affix file.

> what does 'select * from ts_debug('foo');' prints ?

    ts_name   | tok_type | description | token |   dict_name    | tsvector
-------------+----------+-------------+-------+----------------+----------
  utf8_german | lword    | Latin word  | foo   | {de_ispell,de} | 'foo'


Thanks for your time.


--
Regards,
Hannes Dorbath