Re: dynamically allocating chunks from shared memory

Hello Alvaro,

thank you for looking through this code.

On 07/20/2010 07:50 PM, Alvaro Herrera wrote:
> Interesting, thanks.
>
> I gave it a skim and found that it badly needs a lot more code comments.

Hm.. yeah, the dynshmem stuff could probably need more comments. (The 
bgworker stuff is probably a better example).

> I'm also unconvinced that spinlocks are the best locking primitive here.
> Why not lwlocks?

It's derived from a completely lock-free algorithm, as proposed by Maged 
M. Michael in: Scalable Lock-Free Dynamic Memory Allocator. I dropped 
all of the CAS primitives with their retry loop around and did further 
simplifications. Spinlocks simply looked like the simplest thing to 
fall-back to. But yeah, splitting into read and write accesses and using 
lwlocks might be a win. Or it might not. I honestly don't know. And it's 
probably not the best performing allocator ever. But it's certainly 
better than nothing.

I did recently release the lock-free variant as well as a lock based 
one, see http://www.bluegap.ch/projects/wamalloc/ for more information.

> I'm not sure what kind of resistance you'll see to the idea of a
> dynamically allocatable shmem area.

So far neither resistance nor applause. I'd love to hear more of an 
echo. Even if it's resistance.

> Maybe we could use this in other
> areas

..which is why I've published this separately from Postgres-R.

> such as allocating space for heavyweight lock objects.  Right now
> the memory usage for them could grow due to a transitory increase in
> lock traffic, leading to out-of-memory conditions later in other
> modules.  We've seen reports of that problem, so it'd be nice to be able
> to fix that with this infrastructure.

Maybe, yes. Sounds like a nice idea.

> I didn't look at the imessages patch (except to notice that I didn't
> very much like the handling of out-of-memory, but you already knew that).

As all of the allocation problem has now been ripped out, the imessages 
patch got quite a bit smaller. imsg.c now consists of only around 370 
lines of code.

The handling of out-of-(shared)-memory situation could certainly be 
improved, yes. Note that I've already separated out a 
IMessageCreateInternal() method, which simply returns NULL in that case. 
Is that the API you'd prefer?

Getting back to the dynshmem stuff: I don't mind much *which* allocator 
to use. I also looked at jemalloc, but haven't been able to integrate it 
into Postgres. So I've extended my experiment with wamalloc and turned 
it into something usable for Postgres.

Regards

Markus Wanner