Short benchmark runs are bad if the runs aren't long enough to produce consistent results.
Having to do long runs because a benchmarking tool 'converges to reality' over time in reporting a tps number, due to miscalculation, is also bad.
I want to measure TPS at a particular connection count. A fully cached Select Only pgbench produces fairly consistent numbers over short runs of a few minutes.
pgbench's "including connections establishing" number is polluted by fact that for many seconds the benchmark is running with less than the expected number of connections. I thought that was why the 'excluding' number was also printed and I had been relying on that number.
pgbench's "excluding connections establishing" number seems to be a total garbage number which can be way way over the actual tps. During a period when I had a bug causing slow connections I noticed a consistent value of about 100K tps over the measurement intervals. At the end of a 5 minute run it reported 450K tps! There was no point anywhere during the benchmark that it ran anywhere near that number.
I had been using 'excluding' because it 'seemed' perhaps right in the past. It was only when I got crazy numbers I looked at the calculation to find:
tps_exclude = total->cnt / (time_include - (INSTR_TIME_GET_DOUBLE(conn_total_time) / nclients));
The 'cnt' is the total across the entire run including the period when connections are ramping up. I don't see how dividing by the total time minus the average connection time produces the correct result.
Even without buggy slow connections, when connecting 1000 clients, I've wondered why the 'excluding' number seemed a bit higher than any given reporting interval numbers, over a 5 minute run. I now understand why. NOTE: When the system hits 100% cpu utilization(after about the first 100 connections), on a fully cached Select only pgbench, further connections can struggle to get connected which really skews the results.
How about a patch which offered the option to wait on an advisory lock as a mechanism to let the main thread delay the start of the workload after all clients have connected and entered a READY state? This would produce a much cleaner number.