Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

<div class="moz-cite-prefix">On 01/04/2014 12:05 PM, David Fetter wrote:<br /></div><blockquote
cite="mid:20140104080511.GA12040@fetter.org"type="cite"><pre wrap="">I'm sorry I misunderstood about the extension you
wrote.

Is there some way not to use shared memory for it?
</pre></blockquote><br /> No, IMCS ("<font color="#ff0000">In-Memory </font>Columnar Store") is storing data in shared
memory.<br/> Certainly I could allocate shared memory myself, but due to portability and easy maintenance reasons I
decidedto reuse PostgreSQL mechanism of shared memory. The only requirement is that IMSC extension (as well as
pg_stat_statementsextension) should be included in <br /> "shared_preload_libraries" list in postgresql.conf.<br /><br
/>IMCS memory is not somehow interleave with shared memory used for PostgreSQL shared buffers.<br /> And the only
limitationis this 2567Gb limit at Linux, which can be resolved using the patch included in IMCS distributive.<br /><br
/><br/><br /><br /><blockquote cite="mid:20140104080511.GA12040@fetter.org" type="cite"><pre wrap="">
 
Cheers,
David.

On Sat, Jan 04, 2014 at 11:46:25AM +0400, knizhnik wrote:
</pre><blockquote type="cite"><pre wrap="">Hi David,

Sorry, but I do not completely understand your suggestions:

1. IMCS really contains single patch file sysv_shmem.patch.
Applying this patch is not mandatory for using IMCS: it just solves
the problem with support of > 256Gb of shared memory.
Right now PostgreSQL is not able to use more than 256Gb shared
buffers at Linux with standard 4kb pages.
I have found proposal for using MAP_HUGETLB flag in commit fest:

<a class="moz-txt-link-freetext"
href="http://www.postgresql.org/message-id/20131125032920.GA23793@toroid.org">http://www.postgresql.org/message-id/20131125032920.GA23793@toroid.org</a>

but unfortunately it was rejected. Hugepages are intensively used by
Oracle and I think that them will be useful for improving
performance of PorstreSQL. So not just IMCS can benefit from this
patch. My patch  is much more simple - I specially limited scope of
this patch to one file. Certainly switch huge tlb on/off should be
done through postgresql.conf configuration file.

In any case - IMCS can be used without this patch: you just could
not use more than 256Gb memory, even if your system has more RAM.

2. I do not understand "The add-on is not formatted as an EXTENSION"
IMCS was created as standard extension - I just look at the examples
of other PostgreSQL extensions included in PostgreSQL distribution
(for example pg_stat_statements). It can be added using "create
extension imcs" and removed "drop extension imcs" commands.

If there are some violations of PostgreSQL extensions rules, please
let me know, I will fix them.
But I thought that I have done everything in legal way.






On 01/04/2014 03:21 AM, David Fetter wrote:
</pre><blockquote type="cite"><pre wrap="">On Thu, Jan 02, 2014 at 08:48:24PM +0400, knizhnik wrote:
</pre><blockquote type="cite"><pre wrap="">I want to announce implementation of In-Memory Columnar Store
extension for PostgreSQL.
Vertical representation of data is stored in PostgreSQL shared memory.
</pre></blockquote><pre wrap="">Thanks for the hard work!

I noticed a couple of things about this that probably need some
improvement.

1.  There are unexplained patches against other parts of PostgreSQL,
which means that they may break other parts of PostgreSQL in equally
inexplicable ways.  Please rearrange the patch so it doesn't require
this.  This leads to:

2.  The add-on is not formatted as an EXTENSION, which would allow
people to add it or remove it cleanly.

Would you be so kind as to fix these?

Cheers,
David.
</pre></blockquote></blockquote><pre wrap="">
</pre></blockquote><br />