On 2016-04-07 11:56:12 +0200, Fabien COELHO wrote:
> (2) runs which really vary from one to the next, so as
> to have an idea about how much it may vary, what is the
> performance stability.
I don't think this POV makes all that much sense. If you do something
non-comparable, then the results aren't, uh, comparable. Which also
means there's a lower chance to reproduce observed problems.
> Currently pgbench focusses on (2), which may or may not be fine depending on
> what you are doing. From a personal point of view I think that (2) is more
> significant to collect performance data, even if the results are more
> unstable: that simply reflects reality and its intrinsic variations, so I'm
> fine that as the default.
Uh, and what's the benefit of that variability? pgbench isn't a reality
simulation tool, it's a benchmarking tool. And benchmarks with intrisinc
variability are bad benchmarks.
Greetings,
Andres Freund