On 01/04/2014 11:11 PM, Tom Lane wrote:
> knizhnik <knizhnik@garret.ru> writes:
>> On 01/04/2014 12:05 PM, David Fetter wrote:
>>> Is there some way not to use shared memory for it?
>> No, IMCS ("In-Memory Columnar Store") is storing data in shared memory.
> It would probably be better if it made use of the dynamic shared memory
> features that exist in HEAD.
>
> regards, tom lane
Thank you, I will try it.
But I have some concerns:
1. I want IMCS to work with PostgreSQL versions not supporting DSM
(dynamic shared memory), like 9.2, 9.3.1,...
2. IMCS is using PostgreSQL hash table implementation (ShmemInitHash,
hash_search,...)
May be I missed something - I just noticed DSM and have no chance to
investigate it, but looks like hash table can not be allocated in DSM...
3. IMCS is allocating memory using ShmemAlloc. In case of using DSM I
have to provide own allocator (although creation of non-releasing memory
allocator should not be a big issue).
4. Current implementation of DSM still suffers from 256Gb problem.
Certainly I can create multiple segments and so provide workaround
without using huge pages, but it complicates allocator.
5. I wonder if I dynamically add new DSM segment - will it be available
for other PostgreSQL processes? For example I run query which loads data
in IMCS and so needs more space and allocates new DSM segment. Then
another query is executed by other PostgreSQL process which tries to
access this data. This process is not forked from the process created
this new DSM segment, so I do not understand how this segment will be
mapped to the address space of this process, preserving address...
Certainly I can prohibit dynamic extension of IMCS storage (hoping that
in this case there will be no such problem with DSM). But in this case
we will loose the main advantage of using DSM instead of old schema of
plugin's private shared memory.
6. IMCS has some configuration parameters which has to be set through
postgresql.conf. So in any case user has to edit postgresql.conf file.
In case of using DSM it will be not necessary to add IMCS to
shared_preload_libraries list. But I do not think that it is so
restrictive and critical requirement, is it?