On Wed, Dec 7, 2011 at 03:19, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> The results are...taking the median value of each set of runs as
> representative, my patch appears to run marginally faster than head.
> Of course, there is no reason to believe that it should, and I'm
> certain that the difference can be explained by noise, even though
> I've naturally strived to minimise noise.
You should use the t-test to distinguish whether two data sets show a
consistent difference or whether it's just noise. Excel/OpenOffice
have the TTEST() macro for this purpose. For statistics doofuses like
me, just pick mode=2 and type=3 as that's the most conservative.
If the TTEST result is less than 0.05 then you have 95% certainty that
the two dataset are consistently different. If not, you need more
consistent data.
More information here:
http://www.gifted.uconn.edu/siegle/research/t-test/t-test.html
Regards,
Marti