Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On 01/08/2014 10:51 PM, Robert Haas wrote:
> On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> On Mon, Jan 6, 2014 at 4:04 PM, james <james@mansionfamily.plus.com> wrote:
>>>> The point remains that you need to duplicate it into every process that
>>>> might
>>>> want to use it subsequently, so it makes sense to DuplicateHandle into the
>>>> parent, and then to advertise that  handle value publicly so that other
>>>> child
>>>> processes can DuplicateHandle it back into their own process.
>>> Well, right now we just reopen the same object from all of the
>>> processes, which seems to work fine and doesn't require any of this
>>> complexity.  The only problem I don't know how to solve is how to make
>>> a segment stick around for the whole postmaster lifetime.  If
>>> duplicating the handle into the postmaster without its knowledge gets
>>> us there, it may be worth considering, but that doesn't seem like a
>>> good reason to rework the rest of the existing mechanism.
>> I think one has to try this to see if it works as per the need. If it's not
>> urgent, I can try this early next week?
> Anything we want to get into 9.4 has to be submitted by next Tuesday,
> but I don't know that we're going to get this into 9.4.
>
I wonder what is the intended use case of dynamic shared memory?
Is is primarly oriented on PostgreSQL extensions or it will be used also 
in PosatgreSQL core?
In case of extensions, shared memory may be needed to store some 
collected/calculated information which will be used by extension functions.

The main advantage of DSM (from my point of view) comparing with existed 
mechanism of preloaded extension is that it is not necessary to restart 
server to add new extension requiring shared memory.
DSM segment can be attached or created by _PG_init function of the 
loaded module.
But there will be not so much sense in this mechanism if this segment 
will be deleted when there are no more processes attached to it.
So to make DSM really useful for extension it needs some mechanism to 
pin segment in memory during all server/extension lifetime.

May be I am wrong, but I do not see some reasons for creating multiple 
DSM segments by the same extension.
And total number of DSM segments is expected to be not very large (<10). 
The same is true for synchronization primitives (LWLocks for example) 
needed to synchronize access to this DSM segments. So I am not sure if 
possibility to place locks in DSM is really so critical...
We can just reserved some space for LWLocks which can be used by 
extension, so that LWLockAssign() can be used without 
RequestAddinLWLocks or RequestAddinLWLocks can be used not only from 
preloaded extension.

IMHO the main trouble with DSM is lack of guarantee that segment is 
always mapped to the same virtual address.
Without such guarantee it is not possible to use direct (normal) 
pointers inside DSM.
But there seems to be no reasonable solution.