Re: Copy data to DSA area

Hello.

At Wed, 28 Nov 2018 05:13:26 +0000, "Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com> wrote in
<4E72940DA2BF16479384A86D54D0988A6F3BD73A@G01JPEXMBKW04>
> Hi
> 
> >From: Thomas Munro [mailto:thomas.munro@enterprisedb.com]
> >Here we need something better, because once this new kind of memory fills up with
> >leaked objects, you have to restart the whole cluster to recover.
> 
> Thank you for the clarification. I agree that leaving memory leaking in shared memory 
> without freeing it ends up cluster-wide problem. 

If some of the "leaked" objects are liked each other with other
"live" objects including those on local memory, we must tell
whether each "maybe-leaked" object is actually a leak or
not. That looks apprently stupid. Maybe we shouldn't manipulate
objects on shared memory at such a low-level interface. We would
need a kind of resource manager for such structures holding
shared objects and should (perhaps explicitly) register the
objects that shuold be free'd on error-exiting from a scope. This
seems a bit different from reset callbacks of the current
MemoryContext.

> >If the solution involves significantly different control flows (like PG_TRY(),
> >PG_FINALLY() manual cleanup all over the place), then lots of existing palloc-based
> >code will need to be reviewed and rewritten very carefully for use in this new kind of
> >memory context.  If it can be done automagically, then more of our existing code will
> >be able to participate in the construction of stuff that we want to put in shared
> >memory.
> 
> About resetting X->a I was thinking about how to treat dangling pointer inside a new memory
> context machinery instead of using PG_TRY() stuff. That's because putting PG_TRY() all over the 
> place becomes responsible for a developer trying to shared-memory-version MemoryContext and 
> it needs a lot of efforts as you noted. But in my mind even if PG_TRY() is used, it only affects new
> code uses shared-memory version MemoryContext and doesn't affect current code.
> I may be missing something here...

Thomas mentioned exiting upper-layer library code using palloc,
like pg_list.  As I wrote above, pg_list is one of such feature
that would need to be carefully adjusted.

> >I first thought about this in the context of shared plan caches, a topic that appears on
> >this mailing list from time to time.  There are some other fun problems, like cache
> >invalidation for shared caches, space recycling (LRUs etc), and of course
> >locking/concurrency
> >(dsa_allocate() and dsa_free() are concurrency safe, but whatever data structures
> >you build with the memory of course need their own plan for dealing with concurrency).
> >But a strategy for clean-up on errors seems to be a major subproblem to deal with
> >early in the project.  I will be very happy if we can come up with something :-)
> 
> Yeah, I hope we can do it somehow.

In addition to recycling issue within a fixed area, I suppose
that we will need to "expand" the fixed-address shared
memory. PROT_NONE could work for that but it must break some
platforms..

> >On Wed, Nov 14, 2018 at 12:40 AM Ideriha, Takeshi <ideriha.takeshi@jp.fujitsu.com>
> >wrote:
> >> From: Thomas Munro [mailto:thomas.munro@enterprisedb.com]
> >> >postgres=# call hoge_list(3);
> >> >NOTICE:  Contents of list:
> >> >NOTICE:   1
> >> >NOTICE:   2
> >> >NOTICE:   3
> >> >CALL
> >> >
> >> >You can call that procedure from various different backends to add
> >> >and remove integers from a List.  The point is that it could just as
> >> >easily be a query plan or any other palloc-based stuff in our tree, and the pointers
> >work in any backend.
> >>
> >> Thanks for the patch. I know it's a rough sketch but basically that's what I want to
> >do.
> >> I tried it and it works well.
> >
> >After your message I couldn't resist a small experiment.  But it seems there are
> >plenty more problems to be solved to make it useful...
> 
> Thank you for kind remark. 
> By the way I just thought meta variable "hoge" is used only in Japan :)

I've seen it many-many times including mine in this list, thus
why don't you know it is already a world-wide jargon? :p

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center