On Thu, Feb 21, 2019 at 8:28 AM Arthur Zakirov <a.zakirov@postgrespro.ru> wrote:
> Your approach looks simpler. It is necessary just to periodically scan
> dictionaries' cache hash table and not call dsm_pin_segment() when a DSM
> segment initialized. It also means that a dictionary is loaded into DSM
> only while there is a backend which attached the dictionary's DSM.
Right. I think that having a central facility that tries to decide
whether or not a dictionary should be kept in shared memory or not,
e.g. based on a cache size parameter, isn't likely to work well. The
problem is that if we make a decision that a dictionary should be
evicted because it's causing us to exceed the cache size threshold,
then we have no way to implement that decision. We can't force other
backends to remove the mapping immediately, nor can we really bound
the time before they respond to a request to unmap it. They might be
in the middle of using it.
So I think it's better to have each backend locally make a decision
about when that particular backend no longer needs the dictionary, and
then let the system automatically clean up the ones that are needed by
nobody.
Perhaps a better approach still would be to do what Andres proposed
back in March:
#> Is there any chance we can instead can convert dictionaries into a form
#> we can just mmap() into memory? That'd scale a lot higher and more
#> dynamicallly?
The current approach inherently involves double-buffering: you've got
the filesystem cache containing the data read from disk, and then the
DSM containing the converted form of the data. Having something that
you could just mmap() would avoid that, plus it would become a lot
less critical to keep the mappings around. You could probably just
have individual queries mmap() it for as long as they need it and then
tear out the mapping when they finish executing; keeping the mappings
across queries likely wouldn't be too important in this case.
The downside is that you'd probably need to teach resowner.c about
mappings created via mmap() so that you don't leak mappings on an
abort, but that's probably not a crazy difficult problem.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company