Fusion-io ioDrive

I recently got my hands on a device called ioDrive from a company
called Fusion-io.  The ioDrive is essentially 80GB of flash on a PCI
card.  It has its own driver for Linux completely outside of the
normal scsi/sata/sas/fc block device stack, but from the user's
perspective it behaves like a block device.  I put the ioDrive in an
ordinary PC with 1GB of memory, a single 2.2GHz AMD CPU, and an
existing Areca RAID with 6 SATA disks and a 128MB cache.  I tested the
device with PostgreSQL 8.3.3 on Centos 5.3 x86_64 (Linux 2.6.18).

The pgbench database was initialized with scale factor 100.  Test runs
were performed with 8 parallel connections (-c 8), both read-only (-S)
and read-write.  PostgreSQL itself was configured with 256MB of shared
buffers and 32 checkpoint segments.  Otherwise the configuration was
all defaults.

In the following table, the "RAID" configuration has the xlogs on a
RAID 0 of 2 10krpm disks with ext2, and the heap is on a RAID 0 of 4
7200rpm disks with ext3.  The "Fusion" configuration has everything on
the ioDrive with xfs.  I tried the ioDrive with ext2 and ext3 but it
didn't seem to make any difference.

                            Service Time Percentile, millis
        R/W TPS   R-O TPS      50th   80th   90th   95th
RAID      182       673         18     32     42     64
Fusion    971      4792          8      9     10     11

Basically the ioDrive is smoking the RAID.  The only real problem with
this benchmark is that the machine became CPU-limited rather quickly.
During the runs with the ioDrive, iowait was pretty well zero, with
user CPU being about 75% and system getting about 20%.

Now, I will say a couple of other things.  The Linux driver for this
piece of hardware is pretty dodgy.  Sub-alpha quality actually.  But
they seem to be working on it.  Also there's no driver for
OpenSolaris, Mac OS X, or Windows right now.  In fact there's not even
anything available for Debian or other respectable Linux distros, only
Red Hat and its clones.  The other problem is the 80GB model is too
small to hold my entire DB, Although it could be used as a tablespace
for some critical tables.  But hey, it's fast.

I'm going to put this board into my 8-way Xeon to see if it goes any
faster with more CPU available.

I'd be interested in hearing experiences with other flash storage
devices, SSDs, and that type of thing.  So far, this is the fastest
hardware I've seen for the price.

-jwb