Обсуждение: Performance Farm Release
<div class="WordSection1"><p class="MsoNormal">Hey all,<p class="MsoNormal"> <p class="MsoNormal">This is just my email tonotify you all that the project I’ve been working on with Stephen, the PostgreSQL Performance Farm, has been released.As of now, it only supports 9.0, due to the use of workers. More details can be found in the readme. The Git repositoryis located here: http://github.com/slux/Postgre-Performance-Farm<p class="MsoNormal"> <p class="MsoNormal">ScottLuxenberg<p class="MsoNormal">703-610-1823 (W)<p class="MsoNormal">703-303-5189 (C)<p class="MsoNormal">Scott.Luxenberg@noblis.org<pclass="MsoNormal"> </div>
[resending after noticing that "reply all" resulted in sending to pgsql-hackers-owner rather than pgsql-hackers] "Luxenberg, Scott I." <Scott.Luxenberg@noblis.org> wrote: > This is just my email to notify you all that the project I've been > working on with Stephen, the PostgreSQL Performance Farm, has been > released. As of now, it only supports 9.0, due to the use of > workers. More details can be found in the readme. The Git > repository is located here: > http://github.com/slux/Postgre-Performance-Farm If this is covered in a file that I missed, or on a Wiki somewhere, please point me in the right direction and accept my apologies. I looked at the README and didn't find enough to really answer my questions. Would this project be useful for someone trying to assess the performance impact of a proposed patch (at least on the developer's machine)? What would someone do to use it in this way? Thanks for any info. -Kevin
Kevin,
* Kevin Grittner (Kevin.Grittner@wicourts.gov) wrote:
> Would this project be useful for someone trying to assess the
> performance impact of a proposed patch (at least on the developer's
> machine)? What would someone do to use it in this way?
The goal is to have this running in a similar manner as the build farm
to identify when a patch has an impact on performance (good or bad).
Hackers would then be able to view performance farm reports similar to
viewing build farm reports. Not sure if we'd have alerts or something,
but I'd think in alot of cases a given hacker would know that they're
commiting something performance-impacting (or saw someone else commit
something that could be) and they'd go check out the performance farm
reports a few days later to determine if there was a change.
Thanks,
Stephen
>Stephen Frost <sfrost@snowman.net> wrote: > The goal is to have this running in a similar manner as the build > farm to identify when a patch has an impact on performance (good > or bad). Hackers would then be able to view performance farm > reports similar to viewing build farm reports. Not sure if we'd > have alerts or something, but I'd think in alot of cases a given > hacker would know that they're commiting something performance- > impacting (or saw someone else commit something that could be) and > they'd go check out the performance farm reports a few days later > to determine if there was a change. I actually understood that part, but was already wondering if it could be bent to slightly different purposes. It seems as though there would be value to using it to evaluate the performance impact of a proposed patch, at least on a limited basis, *before* a commit. If there's not an immediately obvious way to put it to that alternative use, that's OK; I just thought I'd ask. -Kevin
* Kevin Grittner (Kevin.Grittner@wicourts.gov) wrote: > I actually understood that part, but was already wondering if it > could be bent to slightly different purposes. It seems as though > there would be value to using it to evaluate the performance impact > of a proposed patch, at least on a limited basis, *before* a commit. > If there's not an immediately obvious way to put it to that > alternative use, that's OK; I just thought I'd ask. You can certainly run it yourself locally w/o setting it up to report back to the build or performance farm.. So, yes, you can, you'll just have to look through the outputs yourself and it won't necessairly make much sense unless you've been doing those runs for a period of time to get a feel for how volatile the speed is on your system.. I guess one issue would be figuring out how to inject your patch into the process.. If you have it committed to a git instance somewhere, you could tell it to pull from that after running on the main PG w/o the patch.. Stephen
(resending as I also accidentally CC'd pgsql-hackers-owner, not the list) On 25/08/10 02:25, Luxenberg, Scott I. wrote: > This is just my email to notify you all that the project I've been > working on with Stephen, the PostgreSQL Performance Farm, has been > released. As of now, it only supports 9.0, due to the use of workers. > More details can be found in the readme. The Git repository is located > here: http://github.com/slux/Postgre-Performance-Farm Umm, how about renaming it to "PostgreSQL-Performance-Farm" before the name gets too engrained everywhere and impossible to change anymore? The project is not called "Postgre".. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Right, sorry about that, one of those stupid things I did on a lack of sleep. The github is now here: http://github.com/slux/PostgreSQL-Performance-Farm -----Original Message----- From: Heikki Linnakangas [mailto:heikki.linnakangas@enterprisedb.com] Sent: Wednesday, August 25, 2010 2:31 PM To: Luxenberg, Scott I. Cc: PostgreSQL-development Subject: Re: [HACKERS] Performance Farm Release (resending as I also accidentally CC'd pgsql-hackers-owner, not the list) On 25/08/10 02:25, Luxenberg, Scott I. wrote: > This is just my email to notify you all that the project I've been > working on with Stephen, the PostgreSQL Performance Farm, has been > released. As of now, it only supports 9.0, due to the use of workers. > More details can be found in the readme. The Git repository is located > here: http://github.com/slux/Postgre-Performance-Farm Umm, how about renaming it to "PostgreSQL-Performance-Farm" before the name gets too engrained everywhere and impossible to change anymore? The project is not called "Postgre".. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Kevin Grittner wrote: > I actually understood that part, but was already wondering if it > could be bent to slightly different purposes. It seems as though > there would be value to using it to evaluate the performance impact > of a proposed patch, at least on a limited basis, *before* a commit Eventual design here presumed that this would reach the point where one of the sources you might want to pull from to test performance of was a different git branch than the main one. That's what I was planning to do with it. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.us
Stephen Frost wrote: > You can certainly run it yourself locally w/o setting it up to report > back to the build or performance farm.. So, yes, you can, you'll just > have to look through the outputs yourself and it won't necessairly make > much sense unless you've been doing those runs for a period of time to > get a feel for how volatile the speed is on your system.. > Thanks again to Scott for working on this all summer, and to Stephen and Noblis for helping fund it. There were a lot of small pieces that needed to assemble just right to make this all fit together in a way I hope will make it integrate into the core infrastructure soon. It's probably not obvious to everyone else where this stands as far as reporting results goes though, so let me expand on what Stephen said here. When you run performance tests with this, those are stored locally. Scott demonstrated that you can get basic reports out of that, and I'm content the right initial tests are being run and the most useful data to record is being saved. But none of the performance numbers are sent anywhere yet--they're just stored as CSV files. From the README: "WHENEVER YOU RUN A PERFORMANCE TEST, RUN WITH --test, so not to spam the buildfarm server". That's the bad news that warning is conveying: you can't upload results yet. I expect the next steps here to look like this: 1) Nail down what subset of the information gathered locally should be uploaded to the buildfarm master server. Probably just the same columns of data already being saved for each test, but perhaps with some extra metadata. The local animal will also have graphs and such, but unrealistic to upload all of those for a number of reasons I think. Only really useful to drill down when there is some sort of regression I expect, and hopefully the animal is still alive when that happens. 2) Update the buildfarm server code to accept and store that data. 3) Update this perffarm client to talk to that. 4) Merge the perfarm fork changes into the mainline buildfarm code. I expect continued bitrot of this code as changes are made to the regular buildfarm client, so it might be worth considering that sooner rather than later. My understanding is that the code for the server side of the buildfarm isn't public to everyone right now, just because of time limitations getting it cleaned up for that. So a couple of parts here are being funneled through how much spare time Andrew has, and there are more important git and buildfarm related things for him to worry about right now. Presuming nothing exciting on this happens before then, I'm hoping that I can catch up with Andrew at PG West and map out how to get the rest of this done, so it goes live somewhere during 9.1 development. Now that the code has been released from the Noblis fortress, I can start cleaning up some of the little details on it before then too (i.e. not working for anything but 9.0 yet). -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.us
On Tue, 2010-08-31 at 03:28 -0400, Greg Smith wrote: > 4) Merge the perfarm fork changes into the mainline buildfarm code. I > expect continued bitrot of this code as changes are made to the regular > buildfarm client, so it might be worth considering that sooner rather > than later. As Andrew is currently on vacation, *now* is the time to do that. JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt
On Tue, 2010-08-31 at 03:28 -0400, Greg Smith wrote: > 4) Merge the perfarm fork changes into the mainline buildfarm code. I > expect continued bitrot of this code as changes are made to the regular > buildfarm client, so it might be worth considering that sooner rather > than later. As Andrew is currently on vacation, *now* is the time to do that. JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt
On 08/31/2010 03:28 AM, Greg Smith wrote: > > 1) Nail down what subset of the information gathered locally should be > uploaded to the buildfarm master server. Probably just the same > columns of data already being saved for each test, but perhaps with > some extra metadata. The local animal will also have graphs and such, > but unrealistic to upload all of those for a number of reasons I > think. Only really useful to drill down when there is some sort of > regression I expect, and hopefully the animal is still alive when that > happens. > > 2) Update the buildfarm server code to accept and store that data. The server code is in fact very generic. There is no hard coding of step names or anything like that. The major piece I think we'd need to add in here is somewhere to store performance test scores, as opposed to the success/failure of normal buildfarm steps. That's not going to be terrible difficult. > > 3) Update this perffarm client to talk to that. > > 4) Merge the perfarm fork changes into the mainline buildfarm code. I > expect continued bitrot of this code as changes are made to the > regular buildfarm client, so it might be worth considering that sooner > rather than later. OK, but the client code is hardly in a state of violent flux. In the last year there have been eight commits, including the git changes. > > My understanding is that the code for the server side of the buildfarm > isn't public to everyone right now, just because of time limitations > getting it cleaned up for that. So a couple of parts here are being > funneled through how much spare time Andrew has, and there are more > important git and buildfarm related things for him to worry about > right now. The git buildfarm changes are pretty much bedded down now, I think. I have broken the back of extensible enums a bit more quickly than I expected, so I might have some flying time in the next week or so to devote to cleaning stuff up in the server code, while taking a break from COPY as a FROM target (which will undoubtedly turn my brain to mush if I work on it for more than a couple of hours at a time). > > Presuming nothing exciting on this happens before then, I'm hoping > that I can catch up with Andrew at PG West and map out how to get the > rest of this done, so it goes live somewhere during 9.1 development. > Now that the code has been released from the Noblis fortress, I can > start cleaning up some of the little details on it before then too > (i.e. not working for anything but 9.0 yet). > Sounds good. cheers andrew
I just dusted off this code and brought it back to current again. Basically a lot of reformatting the new performance farm parts to minimize their diff. Once that was done, all of the other buildfarm client updates since then applied cleanly. The result is now sitting as a fork of Andrew's client code repo at https://github.com/greg2ndQuadrant/client-code , replacing the repo Scott published at https://github.com/slux/PostgreSQL-Performance-Farm ; much easier to avoid future bit-rot with this structure. The main changes made here are now pretty easy to read on github: https://github.com/greg2ndQuadrant/client-code/commit/d0339a59ceb4711a6b042d3f1d053c77f07720f4 and I've attached the code as a patch here. It also uses some scripts from pgbench-tools that aren't the interesting part. I got a nibble from Endpoint at PGEast about interest in hacking the buildfarm server code to add support for a new test type, so this may start moving forward again if that works out. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
Вложения
On Mar 27, 2011, at 3:20 PM, Greg Smith <greg@2ndQuadrant.com> wrote: > I just dusted off this code and brought it back to current again. Basically a lot of reformatting the new performancefarm parts to minimize their diff. Once that was done, all of the other buildfarm client updates since then appliedcleanly. > > The result is now sitting as a fork of Andrew's client code repo at https://github.com/greg2ndQuadrant/client-code , replacingthe repo Scott published at https://github.com/slux/PostgreSQL-Performance-Farm ; much easier to avoid future bit-rotwith this structure. > > The main changes made here are now pretty easy to read on github: https://github.com/greg2ndQuadrant/client-code/commit/d0339a59ceb4711a6b042d3f1d053c77f07720f4and I've attached the codeas a patch here. It also uses some scripts from pgbench-tools that aren't the interesting part. > > I got a nibble from Endpoint at PGEast about interest in hacking the buildfarm server code to add support for a new testtype, so this may start moving forward again if that works out. I don't have anything constructive to add to this conversation, but I think it's awesome that you're working on this! ...Robert