Re: [HACKERS] [hackers]development suggestion needed

xun@cs.ucsb.edu (Xun Cheng) writes:
> I want to experiment with some new fast join algorithms.

Cool.  Welcome aboard!

> Could anyone tell me if
> the directory /docs/pgsql/src/backend/executor is the 
> right place to start

The executor is only half the problem: you must also teach the
planner/optimizer how and when to use the new join type.

Hiroshi Inoue has recently been down this path (adding support
for TID-based scans), and might be able to give you more specific
advice.

> 1. Does postgresql do raw storage device management or it relies
>    on file system? My impression is no raw device. If no,
>    is it difficult to add it and possibly how?

Postgres uses Unix files.  We have avoided raw-device access mostly on
grounds of portability.  To persuade people that such a change should go
into the distribution, you'd need to prove that *significantly* better
performance is obtained with raw access.  I for one don't think it's a
foregone conclusion; Postgres gets considerable benefit from sitting
atop Unix kernel device schedulers and disk buffer caches.

As far as the actual implementation goes, the low level access methods
go through a "storage manager" switch that was intended to allow for
the addition of a new storage manager, such as a raw-device manager.
So you could get a good deal of stuff working by implementing code that
parallels md.c/fd.c.  The main problem at this point is that there is a
fair amount of utility code that goes out and does its own manipulation
of the database file structure.  You'd need to clean that up by pushing
it all down below the storage manager switch (inventing new storage
manager calls as needed).

>    that the available join algos implemented are nested loop
>    join (including index-based), hash join (which one? hybrid),
>    sort-merge join?

Right.  The hash join uses batching if it estimates that the relation
is too large to fit in memory; is that what you call "hybrid"?

> 4. Usually a single sequential pass of a large joining relation
>    is preferred to random access in large join operation.
>    It's mostly because of the current disk access characteristics.
>    Is it possible for me to do some benchmarking about this
>    using postgresql? What I'm actually asking are the issues about 
>    how to control the flow of data form disk to buffers,
>    how to stop file system interference and how to arrange
>    actual data placement on the disk.

You don't get to do either of the latter two unless you write a
raw-device storage manager --- which'd be a fair amount of work
for what might be little return.  Are you convinced that that ought
to be the first thing you work on?  I'd be inclined to think about
join algorithms in the abstract, without trying to control physical
disk placement of the data...
        regards, tom lane