Обсуждение: The need for clustered indexes to boost TPC-V performance

Following the earlier email introducing the TPC-V benchmark, and that we are developing an industry standard benchmarking kit for TPC-V using PostgreSQL, here is a specific performance issue we have run into.

Comparing the table sizes, we are close to 2X larger (more on this in a later note). But the index size is what stands out. Our overall index usage (again, after accounting for different numbers of rows) is 4.8X times larger. 35% of our I/Os are to the index space. I am guessing that the 4.8X ballooning has something to do with this, and that in itself explains a lot about our high I/O rate, as well as higher CPU/tran cycles compared to MS SQL (we are  2.5-3 times slower).

Well, MS SQL used a “clustered index” for CT, i.e., the data is held in the leaf pages of the index B-Tree. The data and index are in one data structure. Once you lookup the index, you also have the data at zero additional cost.

[snip]

Is the PGSQL community willing to invest in a feature that a) has been requested by many others already; and b) can make a huge difference in a benchmark that can lend substantial credibility to PGSQL performance?

Craig Ringer, 04.07.2012 07:43:

> I'm not sure what the best option for getting a 9.2 beta build for Windows is.

Download the ZIP from here:

http://www.enterprisedb.com/products-services-training/pgbindownload

Unzip, initdb, pg_ctl start

Regards
Thomas

On 07/04/2012 03:57 PM, Thomas Kellerer wrote:
> Craig Ringer, 04.07.2012 07:43:
>
>> I'm not sure what the best option for getting a 9.2 beta build for
>> Windows is.
>
> Download the ZIP from here:
>
> http://www.enterprisedb.com/products-services-training/pgbindownload

Gah, I'm blind. I looked at that page twice and failed to see the
entries for the beta. Sorry.

--
Craig Ringer

On Tue, Jul 3, 2012 at 10:43 PM, Craig Ringer <ringerc@ringerc.id.au> wrote:
> On 07/04/2012 07:13 AM, Reza Taheri wrote:
>
> Following the earlier email introducing the TPC-V benchmark, and that we are
> developing an industry standard benchmarking kit for TPC-V using PostgreSQL,
> here is a specific performance issue we have run into.
>
>
> Which version of PostgreSQL are you using?
>
> How has it been tuned beyond the defaults - autovacuum settings,
> shared_buffers, effective_cache_size, WAL settings, etc?
>
> How much RAM is on the blade? What OS and version are on the blade?
>
>
> Comparing the table sizes, we are close to 2X larger (more on this in a
> later note). But the index size is what stands out. Our overall index usage
> (again, after accounting for different numbers of rows) is 4.8X times
> larger. 35% of our I/Os are to the index space. I am guessing that the 4.8X
> ballooning has something to do with this, and that in itself explains a lot
> about our high I/O rate, as well as higher CPU/tran cycles compared to MS
> SQL (we are  2.5-3 times slower).
>
> This is making me wonder about bloat issues and whether proper vacuuming is
> being done. If the visibility map and free space map aren't maintained by
> proper vaccum operation everything gets messy, fast.
>
> Well, MS SQL used a “clustered index” for CT, i.e., the data is held in the
> leaf pages of the index B-Tree. The data and index are in one data
> structure. Once you lookup the index, you also have the data at zero
> additional cost.
>
> [snip]
>
>
>
> Is the PGSQL community willing to invest in a feature that a) has been
> requested by many others already; and b) can make a huge difference in a
> benchmark that can lend substantial credibility to PGSQL performance?
>
>
> while PostgreSQL doesn't support covering indexes or clustered indexes at
> this point, 9.2 has added support for index-only scans, which are a half-way
> point of sorts. See:
>
>   http://rhaas.blogspot.com.au/2011/10/index-only-scans-weve-got-em.html
>   http://rhaas.blogspot.com.au/2010/11/index-only-scans.html
>
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2822fb9337a21f98ac4ce850bb4145acf47ca27
>
> If at all possible please see how your test is affected by this PostgreSQL
> 9.2 enhancement. It should make a big difference, and if it doesn't it's
> important to know why.
>
> (CC'd Robert Haas)
>
> I'm not sure what the best option for getting a 9.2 beta build for Windows
> is.
>
>
> As for the "invest" side - that's really a matter for EnterpriseDB, Command
> Prompt, Red Hat, and the other backers who're employing people to work on
> the DB. Consider asking on pgsql-hackers, too; if nothing else you'll get a
> good explanation of the current state and progress toward clustered indexes.
>
> Some links that may be useful to you are:
>
>   http://wiki.postgresql.org/wiki/Todo
>   Things that it'd be good to support/implement at some point. Surprisingly,
> covering/clustered indexes aren't on there or at least aren't easily found.
> It's certainly a much-desired feature despite its apparent absence from the
> TODO.

I think there is, deservingly, a lot of hesitation to implement a
strictly ordered table construct.  A similar feature that didn't quite
get finished -- but maybe can be beaten into shape -- is the
grouped-index-tuple implementation:

http://community.enterprisedb.com/git/

It is mentioned on the TODO page.  It's under the category that is
perhaps poorly syntactically overloaded in the world "cluster".

--
fdr

On Tue, Jul 3, 2012 at 8:13 PM, Reza Taheri <rtaheri@vmware.com> wrote:
> So I looked more closely at the indexes. I chose the CASH_TRANSACTION
> table since it has a single index, and we can compare it more directly to the
> Dell data. If you look at page 34 of http://bit.ly/QeWXhE, the index size of CT
> is 1,278,720KB for 6,120,529,488 rows. That’s less than one byte of index
> per data row!  How could that be?  Well, MS SQL used a “clustered index”
> for CT, i.e., the data is held in the leaf pages of the index B-Tree.
> The data and index are in one data structure. Once you lookup the index,
> you also have the data at zero additional cost. For PGSQL, we had to create
> a regular index, which took up 55GB. Once you do the math, this works out
> to around 30 bytes per row. I imagine we have the 15-byte key along with a
> couple of 4-byte or 8-byte pointers.
...
> So MS SQL beats PGSQL by a) having a lower I/O rate due to no competition
> for the buffer pool from indexes (except for secondary indexes); and b) by
> getting the data with a free lookup, whereas we have to work our way down
> both the index and the data trees.

15-byte key?

What about not storing the keys, but a hash, for leaf nodes?

Assuming it can be made to work for both "range" and "equality" scans,
holding only hashes on leaf nodes would reduce index size, but how
much?

I think it's doable, and I could come up with a spec if it's worth it.
It would have to scan the heap for only two extra index pages (the
extremes that cannot be ruled out) and hash collisions, which doesn't
seem like a big loss versus the reduced index.

On Wed, Jul 4, 2012 at 1:13 AM, Reza Taheri <rtaheri@vmware.com> wrote:

> Checking online, the subject of clustered indexes for PostgreSQL comes up
> often. PGSQL does have a concept called “clustered table”, which means a
> table has been organized in the order of an index. This would help with
> sequential accesses to a table, but has nothing to do with this problem.
> PGSQL folks sometimes refer to what we want as “integrated index”.

I do understand this correctly that we are speaking about the concept
which is known under the term "index organized table" (IOT) in Oracle
land, correct?

http://docs.oracle.com/cd/E11882_01/server.112/e25789/indexiot.htm#CBBJEBIH

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Hi Daniel,
Yes, it sounds like GIT will take us half the way there by getting rid of much of the index I/O if we cluster the
tables.We can set the fillfactor parameter to keep tables sorted after updates. I am not sure what impact inserts will
havesince the primary key keeps growing with new inserts, so perhaps the table will maintain the cluster order and the
benefitsof GIT for new rows, too. GIT won't save CPU cycles the way a clustered/integrated index would, and actually
addsto the CPU cost since the data page has to be searched for the desired tuple. 

Thanks,
Reza

> -----Original Message-----
> From: Daniel Farina [mailto:daniel@heroku.com]
> Sent: Wednesday, July 04, 2012 6:40 AM
> To: Craig Ringer
> Cc: Reza Taheri; pgsql-performance@postgresql.org; Robert Haas
> Subject: Re: [PERFORM] The need for clustered indexes to boost TPC-V
> performance
>
> On Tue, Jul 3, 2012 at 10:43 PM, Craig Ringer <ringerc@ringerc.id.au> wrote:
> > On 07/04/2012 07:13 AM, Reza Taheri wrote:
> >
> > Following the earlier email introducing the TPC-V benchmark, and that
> > we are developing an industry standard benchmarking kit for TPC-V
> > using PostgreSQL, here is a specific performance issue we have run into.
> >
> >
> > Which version of PostgreSQL are you using?
> >
> > How has it been tuned beyond the defaults - autovacuum settings,
> > shared_buffers, effective_cache_size, WAL settings, etc?
> >
> > How much RAM is on the blade? What OS and version are on the blade?
> >
> >
> > Comparing the table sizes, we are close to 2X larger (more on this in
> > a later note). But the index size is what stands out. Our overall
> > index usage (again, after accounting for different numbers of rows) is
> > 4.8X times larger. 35% of our I/Os are to the index space. I am
> > guessing that the 4.8X ballooning has something to do with this, and
> > that in itself explains a lot about our high I/O rate, as well as
> > higher CPU/tran cycles compared to MS SQL (we are  2.5-3 times slower).
> >
> > This is making me wonder about bloat issues and whether proper
> > vacuuming is being done. If the visibility map and free space map
> > aren't maintained by proper vaccum operation everything gets messy,
> fast.
> >
> > Well, MS SQL used a "clustered index" for CT, i.e., the data is held
> > in the leaf pages of the index B-Tree. The data and index are in one
> > data structure. Once you lookup the index, you also have the data at
> > zero additional cost.
> >
> > [snip]
> >
> >
> >
> > Is the PGSQL community willing to invest in a feature that a) has been
> > requested by many others already; and b) can make a huge difference in
> > a benchmark that can lend substantial credibility to PGSQL performance?
> >
> >
> > while PostgreSQL doesn't support covering indexes or clustered indexes
> > at this point, 9.2 has added support for index-only scans, which are a
> > half-way point of sorts. See:
> >
> >   http://rhaas.blogspot.com.au/2011/10/index-only-scans-weve-got-
> em.html
> >   http://rhaas.blogspot.com.au/2010/11/index-only-scans.html
> >
> > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2822fb9
> > 337a21f98ac4ce850bb4145acf47ca27
> >
> > If at all possible please see how your test is affected by this
> > PostgreSQL
> > 9.2 enhancement. It should make a big difference, and if it doesn't
> > it's important to know why.
> >
> > (CC'd Robert Haas)
> >
> > I'm not sure what the best option for getting a 9.2 beta build for
> > Windows is.
> >
> >
> > As for the "invest" side - that's really a matter for EnterpriseDB,
> > Command Prompt, Red Hat, and the other backers who're employing
> people
> > to work on the DB. Consider asking on pgsql-hackers, too; if nothing
> > else you'll get a good explanation of the current state and progress toward
> clustered indexes.
> >
> > Some links that may be useful to you are:
> >
> >   http://wiki.postgresql.org/wiki/Todo
> >   Things that it'd be good to support/implement at some point.
> > Surprisingly, covering/clustered indexes aren't on there or at least aren't
> easily found.
> > It's certainly a much-desired feature despite its apparent absence
> > from the TODO.
>
> I think there is, deservingly, a lot of hesitation to implement a strictly
> ordered table construct.  A similar feature that didn't quite get finished --
> but maybe can be beaten into shape -- is the grouped-index-tuple
> implementation:
>
> http://community.enterprisedb.com/git/
>
> It is mentioned on the TODO page.  It's under the category that is perhaps
> poorly syntactically overloaded in the world "cluster".
>
> --
> fdr

Hi Robert,
Yes, the same concept. Oracle's IOT feature is used often with TPC benchmarks.

Thanks,
Reza

> -----Original Message-----
> From: Robert Klemme [mailto:shortcutter@googlemail.com]
> Sent: Thursday, July 05, 2012 5:30 AM
> To: Reza Taheri
> Cc: pgsql-performance@postgresql.org; Andy Bond (abond@redhat.com);
> Greg Kopczynski; Jignesh Shah
> Subject: Re: [PERFORM] The need for clustered indexes to boost TPC-V
> performance
>
> On Wed, Jul 4, 2012 at 1:13 AM, Reza Taheri <rtaheri@vmware.com> wrote:
>
> > Checking online, the subject of clustered indexes for PostgreSQL comes
> > up often. PGSQL does have a concept called "clustered table", which
> > means a table has been organized in the order of an index. This would
> > help with sequential accesses to a table, but has nothing to do with this
> problem.
> > PGSQL folks sometimes refer to what we want as "integrated index".
>
> I do understand this correctly that we are speaking about the concept which
> is known under the term "index organized table" (IOT) in Oracle land,
> correct?
>
> http://docs.oracle.com/cd/E11882_01/server.112/e25789/indexiot.htm#CBB
> JEBIH
>
> Kind regards
>
> robert
>
> --
> remember.guy do |as, often| as.you_can - without end
> http://blog.rubybestpractices.com/

On 07/05/2012 03:52 PM, Samuel Gendler wrote:
>
>
> On Thu, Jul 5, 2012 at 1:37 PM, Reza Taheri <rtaheri@vmware.com <mailto:rtaheri@vmware.com>> wrote:
>
>
>     I provided more config details in an earlier email.____
>
>     __ __
>
>
>
> I hate to disagree, but unless I didn't get a message sent to the list, you haven't provided any details about your
postgresqlconfig or otherwise adhered to the guidelines for starting a discussion of a performance problem around here.
I just searched my mailbox and no email from you has any such details.  Several people have asked for them, including
myself. You say you will give any details we want, but this is at least the 3rd or 4th request for such details and
theyhave not yet been forthcoming. 


Reza, I went back and looked myself.  I see no specs on OS, or hardware.... unless you mean this:


> http://bit.ly/QeWXhE. This was run on a similar server, and the database size is close to ours.


You're running on windows then?  Server is 96Gig ram, 8 cores, (dell poweredge T610).
with two powervault MD1120 NAS's?

But then I assume you were not running on that, were you.  You were running vmware on it, probably?


-Andy

On Thu, Jul 5, 2012 at 1:37 PM, Reza Taheri <rtaheri@vmware.com> wrote:

I provided more config details in an earlier email.

 

I hate to disagree, but unless I didn't get a message sent to the list

OK, some config details.

We are using:

 

·         Two blades of an HP BladeSystem c-Class c7000 with 2-socket Intel E5520 (Nehalem-EP) processors and 48GB of memory per blade

o   8 cores, 16 threads per blade

o   48GB of RAM per blade

·         Storage was an EMC VNX5700 with 14 SSDs fronting 32 15K RPM drives

·         The Tier B database VM was alone on a blade with 16 vCPUs, 40GB of memory, 4 virtual drives with various RAID levels

·         The driver and Tier A VMs were on the second blade

o   So we set PGHOST on the client system to point to the server

·         RHEL 6.1

·         PostgreSQL 8.4

·         unixODBC 2.3.2

 

We stuck with PGSQL 8.4 since it is the stock version shipped with RHEL 6. I am building a new, larger testbed, and will switch to PGSQL 9 with that.

 

Well, I keep failing to send an email with an attachment. Do I need a moderator's approval?

Yes, running on VMs and a lower bin processor. With the virtualization overhead, etc., I figure we would be running
rightaround 2/3 of the Dell throughput if we were running the same DBMS. 

I sent the following message twice today with attachments (postgresql.conf, etc.), and it hasn't been posted yet. Here
itis without an attachment. 

****************************************************

From: Reza Taheri
Sent: Thursday, July 05, 2012 11:34 AM
To: 'Craig Ringer'
Cc: pgsql-performance@postgresql.org; Robert Haas
Subject: RE: [PERFORM] The need for clustered indexes to boost TPC-V performance

OK, some config details.
We are using:

*    Two blades of an HP BladeSystem c-Class c7000 with 2-socket Intel E5520 (Nehalem-EP) processors and 48GB of memory
perblade 
o    8 cores, 16 threads per blade
o    48GB of RAM per blade
*    Storage was an EMC VNX5700 with 14 SSDs fronting 32 15K RPM drives
*    The Tier B database VM was alone on a blade with 16 vCPUs, 40GB of memory, 4 virtual drives with various RAID
levels
*    The driver and Tier A VMs were on the second blade
o    So we set PGHOST on the client system to point to the server
*    RHEL 6.1
*    PostgreSQL 8.4
*    unixODBC 2.3.2

We stuck with PGSQL 8.4 since it is the stock version shipped with RHEL 6. I am building a new, larger testbed, and
willswitch to PGSQL 9 with that. 

Postgres.conf is attached.

Thanks,
Reza

> -----Original Message-----
> From: Andy Colson [mailto:andy@squeakycode.net]
> Sent: Thursday, July 05, 2012 5:42 PM
> To: Samuel Gendler
> Cc: Reza Taheri; Robert Klemme; pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] The need for clustered indexes to boost TPC-V
> performance
>
> On 07/05/2012 03:52 PM, Samuel Gendler wrote:
> >
> >
> > On Thu, Jul 5, 2012 at 1:37 PM, Reza Taheri <rtaheri@vmware.com
> <mailto:rtaheri@vmware.com>> wrote:
> >
> >
> >     I provided more config details in an earlier email.____
> >
> >     __ __
> >
> >
> >
> > I hate to disagree, but unless I didn't get a message sent to the list, you
> haven't provided any details about your postgresql config or otherwise
> adhered to the guidelines for starting a discussion of a performance
> problem around here.  I just searched my mailbox and no email from you
> has any such details.  Several people have asked for them, including myself.
> You say you will give any details we want, but this is at least the 3rd or 4th
> request for such details and they have not yet been forthcoming.
>
>
> Reza, I went back and looked myself.  I see no specs on OS, or hardware....
> unless you mean this:
>
>
> > http://bit.ly/QeWXhE. This was run on a similar server, and the database
> size is close to ours.
>
>
> You're running on windows then?  Server is 96Gig ram, 8 cores, (dell
> poweredge T610).
> with two powervault MD1120 NAS's?
>
> But then I assume you were not running on that, were you.  You were
> running vmware on it, probably?
>
>
> -Andy

You're running on windows then?  Server is 96Gig ram, 8 cores, (dell poweredge T610).
with two powervault MD1120 NAS's?

On 07/03/2012 07:13 PM, Reza Taheri wrote:
> Is the PGSQL community willing to invest in a feature that a) has been
> requested by many others already; and b) can make a huge difference in a
> benchmark that can lend substantial credibility to PGSQL performance?

Larger PostgreSQL features usually get built because companies sponsor
their development, they pass review as both useful & correct, and then
get committed.  Asking the community to invest in a new feature isn't
quite the right concept.  Yes, everyone would like one of the smaller
index representations.  I'm sure we can find reviewers willing to look
at such a feature and committers who would also be interested enough to
commit it, on a volunteer basis.  But a feature this size isn't going to
spring to life based just on volunteer work.  The most useful questions
would be "who would be capable of writing that feature?" and "how can we
get them sponsored to focus on it?"  I can tell from your comments yet
what role(s) in that process VMWare wants to take on internally, and
which it's looking for help with.  The job of convincing people it's a
useful feature isn't necessary--we know that's true.

--
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

On 07/06/2012 09:00 AM, Reza Taheri wrote:
> Well, I keep failing to send an email with an attachment. Do I need a moderator's approval?

Probably. If so, it's really annoying that mailman isn't telling you
this via a "held for moderation" auto-reply. It should be.

> We stuck with PGSQL 8.4 since it is the stock version shipped with RHEL 6. I am building a new, larger testbed, and
willswitch to PGSQL 9 with that. 

Just so you know, as per PostgreSQL versioning policy major releases are
x.y, eg "8.4", "9.0" and "9.1" are distinct major releases.

http://www.postgresql.org/support/versioning/

I've always found that pretty odd and wish major releases would just
increment the first version part, but the policy states how it's being
done. It's important to realize this when you're talking about Pg
releases, because 8.4, 9.0, 9.1 and 9.2 are distinct releases with
different feature sets, so "postgresql 9" doesn't mean much.

--
Craig Ringer

Just to be clear, we have a number of people from different companies working on the kit. This is not a VMware project,
itis a TPC project. But I hear you regarding coming in from the cold and asking for a major db engine feature. I know
thatI have caused a lot of rolling eyes. Believe me, I have had the same (no, worse!) reaction from every one of the
commercialdatabase companies in response to similar requests over the past 25 years. 

We have our skin in the game, and as long as the community values the benchmark and wants to support us, we will figure
outthe details as we go forward. 

Thanks,
Reza

> -----Original Message-----
> From: Greg Smith [mailto:greg@2ndQuadrant.com]
> Sent: Thursday, July 05, 2012 6:42 PM
> To: Reza Taheri
> Cc: pgsql-performance@postgresql.org; Andy Bond (abond@redhat.com);
> Greg Kopczynski; Jignesh Shah
> Subject: Re: [PERFORM] The need for clustered indexes to boost TPC-V
> performance
>
> On 07/03/2012 07:13 PM, Reza Taheri wrote:
> > Is the PGSQL community willing to invest in a feature that a) has been
> > requested by many others already; and b) can make a huge difference in
> > a benchmark that can lend substantial credibility to PGSQL performance?
>
> Larger PostgreSQL features usually get built because companies sponsor
> their development, they pass review as both useful & correct, and then get
> committed.  Asking the community to invest in a new feature isn't quite the
> right concept.  Yes, everyone would like one of the smaller index
> representations.  I'm sure we can find reviewers willing to look at such a
> feature and committers who would also be interested enough to commit it,
> on a volunteer basis.  But a feature this size isn't going to spring to life based
> just on volunteer work.  The most useful questions would be "who would
> be capable of writing that feature?" and "how can we get them sponsored
> to focus on it?"  I can tell from your comments yet what role(s) in that
> process VMWare wants to take on internally, and which it's looking for help
> with.  The job of convincing people it's a useful feature isn't necessary--we
> know that's true.
>
> --
> Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

On Thu, Jul 5, 2012 at 10:33 PM, Reza Taheri <rtaheri@vmware.com> wrote:
> Just to be clear, we have a number of people from different companies working on the kit. This is not a VMware
project,it is a TPC project. But I hear you regarding coming in from the cold and asking for a major db engine feature.
Iknow that I have caused a lot of rolling eyes. Believe me, I have had the same (no, worse!) reaction from every one of
thecommercial database companies in response to similar requests over the past 25 years. 

No rolling of eyes from me.  Clustered indexes work and if your table
access mainly hits the table through that index you'll see enormous
reductions in i/o.  Index only scans naturally are a related
optimization in the same vein.  Denying that is just silly.  BTW,
putting postgres through a standard non trivial benchmark suite over
reasonable hardware, reporting results, identifying bottlenecks, etc.
is incredibly useful.  Please keep it up, and don't be afraid to ask
for help here.  (one thing I'd love to see is side by side results
comparing 8.4 to 9.1 to 9.2).

merlin

Hi Merlin,
We are moving up to a larger testbed, and are planning to use 9.2. But the results will not comparable to our 8.4
resultsdue to differences in hardware. But that comparison is a useful one. I'll try for a quick test on the new
hardwarewith 8.4 before moving to 9.2. 

Thanks,
Reza

> -----Original Message-----
> From: Merlin Moncure [mailto:mmoncure@gmail.com]
> Sent: Tuesday, July 10, 2012 12:06 PM
> To: Reza Taheri
> Cc: Greg Smith; pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] The need for clustered indexes to boost TPC-V
> performance
>
> On Thu, Jul 5, 2012 at 10:33 PM, Reza Taheri <rtaheri@vmware.com> wrote:
> > Just to be clear, we have a number of people from different companies
> working on the kit. This is not a VMware project, it is a TPC project. But I
> hear you regarding coming in from the cold and asking for a major db engine
> feature. I know that I have caused a lot of rolling eyes. Believe me, I have
> had the same (no, worse!) reaction from every one of the commercial
> database companies in response to similar requests over the past 25 years.
>
> No rolling of eyes from me.  Clustered indexes work and if your table access
> mainly hits the table through that index you'll see enormous reductions in
> i/o.  Index only scans naturally are a related optimization in the same vein.
> Denying that is just silly.  BTW, putting postgres through a standard non
> trivial benchmark suite over reasonable hardware, reporting results,
> identifying bottlenecks, etc.
> is incredibly useful.  Please keep it up, and don't be afraid to ask for help
> here.  (one thing I'd love to see is side by side results comparing 8.4 to 9.1 to
> 9.2).
>
> merlin