On Tue, Apr 15, 2014 at 5:47 PM, Josh Berkus <josh@agliodbs.com> wrote:
Hackers,
I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us.
How can we make beta testing better and more effective? How can we get more users to actually throw serious workloads at new versions and share the results?
I've tried a couple of things over the last two years and they haven't worked all that well. Since we're about to go into another beta testing period, we need something new. Ideas?
I think it boils down to making it really easy to create a workload generator. Most companies have simple single-threaded regression tests for functionality but very few companies have good parallel workload generators which reflect activities in their production environment.
A documented beta test process/toolset which does the following would help:
1) Enables full query logging
2) Creates a replica of a production DB, record $TIME when it stops.
3) Allow user to make changes (upgrade to 9.4, change hardware, change kernel settings, ...)
4) Plays queries from the CSV logs starting from $TIME mimicking actual timing and transaction boundaries
If Pg can make it easy to duplicate activities currently going on in production inside another environment, I would be pleased to fire a couple billion queries through it over the next few weeks.
#4 should include reporting useful to the project, such as a sampling of queries which performed significantly worse and a few relative performance stats for overall execution time.