On 12/04/2013 05:40 AM, Peter Eisentraut wrote:
> On Tue, 2013-12-03 at 14:44 -0800, Josh Berkus wrote:
>> Would certainly be nice. Realistically, getting good automated
>> performace tests will require paying someone like Greg S., Mark or me
>> for 6 solid months to develop them, since worthwhile open source
>> performance test platforms currently don't exist. That money has
>> never been available; maybe I should do a kickstarter.
>
> I think the problem is, it's not even clear what the deliverable might
> be. Benchmarking tools exist, and running them on a regular schedule
> shouldn't be difficult. But that doesn't find regressions between
> kernel versions, for example, or regressions in particular queries
> (unless they happen to be included in the benchmark).
I agree on the problem of specifying an exact deliverable - however
simple using some of the extisting benchmark tool and maybe augment them
by the myriad of simple "micro" level regressions we have in the form of
sql queries in the archives would be a sensible start. It might not help
for all cases but it can help for some and we learn something that might
help us building the next iteration of it. Adding say some operatimng
systems to the mix of we have the above would be fairly easy - running a
few kvm instances that get bootstrapped automatically is something that
is a solved problem.
>
> The first step here should be to work out the minimum viable product,
> and then see what it would take to get that done.
yeah we need to start somewhere and see what we can learn.
Stefan