On Tue, Jun 04, 2019 at 01:49:23PM -0400, Stephen Frost wrote:
>Greetings,
>
>* Melanie Plageman (melanieplageman@gmail.com) wrote:
>> Peter and I implemented this small (attached) patch to extend
>> abbreviated key compare sort to macaddr8 datatype (currently supported
>> for macaddr).
>
>Nice.
>
>> I think that that seems like an improvement. I was thinking of
>> registering this patch for the next commitfest. Is that okay?
>
>Sure.
>
>> I was just wondering what the accepted way to test and share
>> performance numbers is for a small patch like this. Is sharing DDL
>> enough? Do I need to use pg_bench?
>
>Detailed (enough... doesn't need to include timing of every indivudal
>query, but something like the average timing across 5 runs or similar
>would be good) results posted to this list, with enough information
>about how to reproduce the tests, would be the way to go.
>
>The idea is to let others also test and make sure that they come up with
>similar results to yours, and if they don't, ideally have enough
>information to narrow down what's happening / what's different.
>
Yeah, there's no "approved way" to do performance tests the contributors
would have to follow. That applies both to tooling and how detailed the
data need/should be. Ultimately, the goal is to convince others (and
yourself) that the change is an improvement. Does a simple pgbench
script achieve that? Cool, use that. Do you need something more complex?
Sure, do a shell script or something like that.
As long as others can reasonably reproduce your tests, it's fine.
For me, the most critical part of benchmarking a change is deciding what
to test - which queries, data sets, what amounts of data, config, etc.
For example, the data set you used has ~12k rows. Does the behavior
change with 10x or 100x that? It probably does not make sense to go
above available RAM (the I/O costs are likely to make everything else
mostly irrelevant), but CPU caches may matter a lot. Different work_mem
(and maintenance_work_mem) values may be useful too.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services