>>>>> "Gavin" == Gavin Flower <GavinFlower@archidevsys.co.nz> writes:
Gavin> What are the standard deviations?
Gavin> Do the arithmetic means change much if you exclude the 2 fastestGavin> & 2 slowest?
Gavin> How do the arithmetic means compare to their respective medians?
Gavin> Essentially, how consistent are the results, or how great is theGavin> noise? There may be better indicators
thanthe ones I'veGavin> suggested above.
This is all rather missing the point.
The relevant metric is not how much noise is introduced between runs of
the same code, but rather how much noise is introduced as a result of
non-consequential changes to the code.
I can get variations of several percent - easily more than three sigmas
of the timing of repeated runs of unchanged code - in the time taken to
sort a float8 column simply from introducing varying amounts of padding
into the body of a function which is never called in the test. Clearly,
the only possible effect here is that the changed memory addresses of
functions must be resulting in different patterns of cache misses /
cache replacements, or TLB misses, or similar low-level effects which
have nothing to do with the code as such.
(That this is a low-level alignment effect is supported by the fact that
the performance changes are not monotonic in the size of the padding;
adding more padding may cause either speedups or slowdowns.)
--
Andrew (irc:RhodiumToad)