Обсуждение: Re: Fork-based version of pgbench

Now that I've fixed the silly mistake in the fork-based version of
pgbench,
http://archives.postgresql.org/pgsql-patches/2005-12/msg00017.php
I'm seeing it consistently outperform the CVS-tip version by about 5%.
I get about 700 tps versus 670 tps; meanwhile "top" reports that idle
CPU percentage drops from around 15% to around 5%.  So I'm thinking
it may be worthwhile to fix the portability issues (ie, provide a
thread-based variant for Windows) and make it the standard version.
It's not so much that I want to inflate the measurements, as that
leaving 10% of the CPU on the table reduces pgbench's usefulness as
a way of stress-testing the backend.

The test case I'm looking at is on a dual Xeon, EM64T, hyperthreading
enabled (hence, 4 logical CPUs), running Fedora Core 4.  Database
parameters are stock except these changes to minimize I/O:fsync = offshared_buffers = 50000checkpoint_segments = 30
Database is initialized with "pgbench -i -s 10 bench" and then tested
with "pgbench -c 10 -t 3000 bench"; I usually do three runs and take
the median to have a trustworthy number.

It'd be interesting to find out if other people can get similar results
on other platforms.
        regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

> It's not so much that I want to inflate the measurements, as that
> leaving 10% of the CPU on the table reduces pgbench's usefulness as
> a way of stress-testing the backend.

I suspect the difference is the same thing you theorised made the difference
before. Namely that they're no longer proceeding in lockstep. The progress is
more random allowing some processes to make more progress than average and
benefit from better, er, well some cache somewhere.

In any case it seems like there would be cases where each kind of behaviour
would be useful. It seems likely there would be bugs where the lockstep
behaviour was useful for testing, and other bugs where the randomized
behaviour would be useful for testing too.


-- 
greg

Greg Stark <gsstark@mit.edu> writes:
> Tom Lane <tgl@sss.pgh.pa.us> writes:
>> It's not so much that I want to inflate the measurements, as that
>> leaving 10% of the CPU on the table reduces pgbench's usefulness as
>> a way of stress-testing the backend.
> ...
> In any case it seems like there would be cases where each kind of behaviour
> would be useful. It seems likely there would be bugs where the lockstep
> behaviour was useful for testing, and other bugs where the randomized
> behaviour would be useful for testing too.

No, I'm not thinking about bugs at all, but performance stress tests.
As an example, the various context-swap-storm behaviors we've been
chasing over the past year or so cannot arise unless you can get several
processes competing for the same resource at the same time.  If your
test configuration isn't capable of saturating the CPU then you have
a much lower probability of being able to observe such misbehavior.
        regards, tom lane