>>>>> "Tomas" == Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
Tomas> Interesting, but I think Gavin was asking about how muchTomas> variation was there for each tested case (e.g.
queryexecuted onTomas> the same code / dataset). And in those cases the padding /Tomas> alignment won't change, and
thusthe effects you describe won'tTomas> influence the results, no?
My point is exactly the fact that since the result is not affected, this
variation between runs of the same code is of no real relevance to the
question of whether a given change in performance can properly be
attributed to a patch.
Put it this way: suppose I have a test that when run repeatedly with no
code changes takes 6.10s (s=0.025s), and I apply a patch that changes
that to 6.26s (s=0.025s). Did the patch have an impact on performance?
Now suppose that instead of applying the patch I insert random amounts
of padding in an unused function and find that my same test now takes a
mean of 6.20s (s=0.058s) when I take the best timing for each padding
size and calculate stats across sizes. Now it looks obvious that the
actual code of the patch probably wasn't responsible for any change...
The numbers used here aren't theoretical; they are obtained by testing a
single query - "select * from d_flt order by v offset 10000000" where
d_flt contains 5 million float8 values - over 990 times with 33
different random padding sizes (uniform in 0-32767). Here's a scatter
plot, with 3 runs of each padding size so you can see the repeatability:
http://tinyurl.com/op9qg8a
--
Andrew.