Re: In-Memory Columnar Store

<div dir="ltr"><br /><div class="gmail_extra"><br /><br /><div class="gmail_quote">2013/12/9 knizhnik <span
dir="ltr"><<ahref="mailto:knizhnik@garret.ru" target="_blank">knizhnik@garret.ru</a>></span><br /><blockquote
class="gmail_quote"style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hello!<br
/><br/> I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL:<br /><br />    
 Documentation:<a href="http://www.garret.ru/imcs/user_guide.html"
target="_blank">http://www.garret.ru/imcs/<u></u>user_guide.html</a><br/>      Sources: <a
href="http://www.garret.ru/imcs-1.01.tar.gz"target="_blank">http://www.garret.ru/imcs-1.<u></u>01.tar.gz</a><br /><br
/>Any feedbacks, bug reports and suggestions are welcome.<br /><br /> Vertical representation of data is stored in
PostgreSQLshared memory.<br /> This is why it is important to be able to utilize all available physical memory.<br />
Nowservers with Tb or more RAM are not something exotic, especially in financial world.<br /> But there is limitation
inLinux with standard 4kb pages  for maximal size of mapped memory segment: 256Gb.<br /> It is possible to overcome
thislimitation either by creating multiple segments - but it requires too much changes in PostgreSQL memory manager.<br
/>Or just set MAP_HUGETLB flag (assuming that huge pages were allocated in the system).<br /><br /> I found several
messagesrelated with MAP_HUGETLB flag, the most recent one was from 21 of November:<br /><a
href="http://www.postgresql.org/message-id/20131125032920.GA23793@toroid.org"
target="_blank">http://www.postgresql.org/<u></u>message-id/20131125032920.<u></u>GA23793@toroid.org</a><br/><br /> I
wonderwhat is the current status of this patch?<span class=""><font color="#888888"><br /><br /><br /><br /><br /><br
/><br/> -- <br /> Sent via pgsql-hackers mailing list (<a href="mailto:pgsql-hackers@postgresql.org"
target="_blank">pgsql-hackers@postgresql.org</a>)<br/> To make changes to your subscription:<br /><a
href="http://www.postgresql.org/mailpref/pgsql-hackers"
target="_blank">http://www.postgresql.org/<u></u>mailpref/pgsql-hackers</a><br/></font></span></blockquote></div><br
/><br/></div><div class="gmail_extra">Hello,<br /></div><div class="gmail_extra">           excellent work! I begin to
dotesting and it's very fast, by the way I found a strange case of "endless" query with CPU a 100%  when the value used
asfilter does not exists:<br /><br /></div><div class="gmail_extra">I am testing with postgres 9.3.1 on debian  and I
useddefault value for the extension except memory ( 512mb )<br /><br /></div><div class="gmail_extra">how to recreate
thetest case :<br /></div><div class="gmail_extra"><br /></div><div class="gmail_extra">## create a table :<br /><br
/></div><divclass="gmail_extra">create table endless ( col1 int , col2 char(30) , col3 int ) ;<br /><br /></div><div
class="gmail_extra">##insert some values:<br /></div><div class="gmail_extra"><br />insert into endless values ( 1,
'ahahahaha',3);<br /><br />insert into endless values ( 2, 'ghghghghg', 4);<br /><br />## create the column store
objects:<br/><br />select cs_create('endless','col1','col2');<br />  cs_create <br />-----------<br /> <br />(1 row)<br
/><br/></div><div class="gmail_extra">## try and test column store :<br /></div><div class="gmail_extra"><br />select
cs_avg(col3)from  endless_get('ahahahaha');<br /> cs_avg <br /> --------<br />      3<br />(1 row)<br /><br />select
cs_avg(col3)from  endless_get('ghghghghg');<br /> cs_avg <br />--------<br />      4<br />(1 row)<br /><br /></div><div
class="gmail_extra">##now select with a value that does not exist :<br /></div><div class="gmail_extra"><br />select
cs_avg(col3)from  endless_get('testing');<br /><br /></div><div class="gmail_extra"># and now  start to loop on cpu and
seemsto never ends , I had to terminate backend<br /></div><div class="gmail_extra"><br /></div><div
class="gmail_extra">Bye<br/><br />Mat<br /></div></div>