Re: Performance Farm Release

Stephen Frost wrote:
> You can certainly run it yourself locally w/o setting it up to report
> back to the build or performance farm..  So, yes, you can, you'll just
> have to look through the outputs yourself and it won't necessairly make
> much sense unless you've been doing those runs for a period of time to
> get a feel for how volatile the speed is on your system..
>   

Thanks again to Scott for working on this all summer, and to Stephen and 
Noblis for helping fund it.  There were a lot of small pieces that 
needed to assemble just right to make this all fit together in a way I 
hope will make it integrate into the core infrastructure soon.

It's probably not obvious to everyone else where this stands as far as 
reporting results goes though, so let me expand on what Stephen said 
here.  When you run performance tests with this, those are stored 
locally.  Scott demonstrated that you can get basic reports out of that, 
and I'm content the right initial tests are being run and the most 
useful data to record is being saved.  But none of the performance 
numbers are sent anywhere yet--they're just stored as CSV files.  From 
the README:  "WHENEVER YOU RUN A PERFORMANCE TEST, RUN WITH --test, so 
not to spam the buildfarm server".  That's the bad news that warning is 
conveying:  you can't upload results yet.

I expect the next steps here to look like this:

1) Nail down what subset of the information gathered locally should be 
uploaded to the buildfarm master server.  Probably just the same columns 
of data already being saved for each test, but perhaps with some extra 
metadata.  The local animal will also have graphs and such, but 
unrealistic to upload all of those for a number of reasons I think.  
Only really useful to drill down when there is some sort of regression I 
expect, and hopefully the animal is still alive when that happens.

2) Update the buildfarm server code to accept and store that data.

3) Update this perffarm client to talk to that.

4) Merge the perfarm fork changes into the mainline buildfarm code.  I 
expect continued bitrot of this code as changes are made to the regular 
buildfarm client, so it might be worth considering that sooner rather 
than later.

My understanding is that the code for the server side of the buildfarm 
isn't public to everyone right now, just because of time limitations 
getting it cleaned up for that.  So a couple of parts here are being 
funneled through how much spare time Andrew has, and there are more 
important git and buildfarm related things for him to worry about right now.

Presuming nothing exciting on this happens before then, I'm hoping that 
I can catch up with Andrew at PG West and map out how to get the rest of 
this done, so it goes live somewhere during 9.1 development.  Now that 
the code has been released from the Noblis fortress, I can start 
cleaning up some of the little details on it before then too (i.e. not 
working for anything but 9.0 yet).

-- 
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us