On Thu, 14 Dec 2006, Tom Lane wrote:
> The pgbench app itself becomes the bottleneck at high transaction rates.
> Awhile back I rewrote it to improve its ability to issue commands
> concurrently, but then desisted from submitting the changes --- if we
> change the app like that, future numbers would be incomparable to past
> ones, which sort of defeats the purpose of a benchmark no?
Not at all. Here's an example from the PC hardware benchmarking
landscape. Futuremark Corporation has a very popular benchmark for 3D
hardware called 3DMark. Every year, they release a new version, and
numbers from it are completely different from those produced by the
previous year's version. That lets them rev the entire approach taken by
the benchmark to reflect current practice. So when the standard for
high-end hardware includes, say, acceleration of lighting effects, the new
version will include a lighting test. In order to break 1000 points (say)
on that test, you absolutely have to have lighting acceleration, even
though on the previous year's test you could score that high without it.
That is not an isolated example; every useful PC benchmark gets updated
regularly, completely breaking backward compatibility, to reflect the
capabilities of current hardware and software. Otherwise we'd still be
testing how well DOS runs on new processors. Right now everyone is (or
has already) upgraded their PC benchmarking code such that you need a
dual-core processor to do well on some of the tests.
If you have a pgbench version with better concurrency features, I for one
would love to see it. I'm in the middle of patching that thing up right
now anyway but hadn't gotten that far (yet--I just spent some of yesterday
staring at how it submits into libpq trying to figure out how to improve
that). I would be happy to take your changes, my changes, changes to the
base code since you forked it, and reconcile everything together into a
pgbench2007--whose results can't be directly compared to the earlier
version, but are more useful on current gen multi-processor/core systems.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD