"Greg Smith" <gsmith@gregsmith.com> writes:
> On Sun, 26 Aug 2007, Kevin Grittner wrote:
>
>> I also think we need to somehow develop a set of tests which report maximum
>> response time on (what should be) fast queries while the database is under
>> different loads, so that those of us for whom reliable response time is more
>> important than maximum overall throughput are protected from performance
>> regressions.
>
> My guess is that the DBT2 tests that Heikki has been running are a more
> complicated than you think they are; there are response time guarantee
> requirements in there as well as the throughput numbers. The tests that I run
> (which I haven't been publishing yet but will be with the final patch soon)
> also report worst-case and 90-th percentile latency numbers as well as TPS. A
> "regression" that improved TPS at the expense of those two would not be
> considered an improvement by anyone involved here.
TPCC requires that the 90th percentile response time be under 5s for most
transactions. It also requires that the average be less than the 90th
percentile which helps rule out circumstances where the longest 10% response
times are *much* longer than 5s.
However in practice neither of those requirements really rule out some pretty
bad behaviour as long as it's rare enough. Before the distributed checkpoint
patch went in we were finding 60s of zero activity at every checkpoint. But
there were so few transactions affected that in the big picture it didn't
impact the 90th percentile. It didn't even affect the 95th percentile. I think
you had to look at the 99th percentile before it even began to impact the
results.
I can't really imagine a web site operator being happy if he was told that
only 1% of user's clicks resulted in a browser timeout...
-- Gregory Stark EnterpriseDB http://www.enterprisedb.com