Re: Hash index todo list item

On Sun, Sep 02, 2007 at 10:41:22PM -0400, Tom Lane wrote:
> Kenneth Marshall <ktm@rice.edu> writes:
> > ... This is the rough plan. Does anyone see anything critical that
> > is missing at this point?
> 
> Sounds pretty good.  Let me brain-dump one item on you: one thing that
> hash currently has over btree is the ability to handle index items up
> to a full page.  Now, if you go with a scheme that only stores hash
> codes and not the underlying data, you can not only handle that but
> improve on it; but if you reduce the bucket size and don't remove the
> data, it'd be a step backward.  The idea I had about dealing with that
> was to only reduce the size of primary buckets --- if it's necessary to
> add overflow space to a bucket, the overflow units are still full pages.
> So an index tuple up to a page in size can always be accommodated by
> adding an overflow page to the bucket.
> 
> Just a thought, but AFAIR it's not in the archives anywhere.
> 
>             regards, tom lane
> 
Tom,

Thank you for the input. I agree that keeping the ability to accomodate
an index tuple up to a page is size worth keeping. I think that your
goal in reducing the bucket size is to improve the packing efficiency
of the index. Since the on disk page size remains the same, it may be
possible to use a different structure overlayed on the current bucket
size and still improve the packing efficiency of the index. After some
more mulling, here are some further thoughts on the improved hash table
implementation:

- Hash lookup is O(1) while btree is O(logN). Is there a value in optimizing the NOT case, i.e. the entry is not in the
table?

- Space versus performance trade-off. This may tie into cache efficiency and use of L2/L3, shared buffers, main memory.
Denserlayouts with a higher load facter may be a bit slower in lookups but play much nicer in a multi-user system. Look
atthe possibility of a lossy mapping?
 

- Build time versus update time. How does concurrency enter into the picture regarding simultaneous updates, inserts,
deletes,and lookups?
 

- Could a hybrid structure with some type of prefix compression give a more efficient layout and possibly improve
performance?

- Index larger fields. Btree is limited to blocksize/3, the current hash implementation can go up to a full block.

- What about multi-column indexes? The current implementation only supports 1 column.

More ideas are welcome and I will add them to the list for
investigation.

Regards,
Ken