Обсуждение: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
Thought I'd pass this along, since the Linux vs FreeBSD performance question comes up fairly regularly... BTW, I've already asked about benchmarking with PostgreSQL, so please don't go over there making trouble. :) ----- Forwarded message from Kris Kennaway <kris@obsecurity.org> ----- X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on noel.decibel.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_50, FORGED_RCVD_HELO,SPF_PASS autolearn=no version=3.1.6 Date: Sat, 24 Feb 2007 16:31:11 -0500 From: Kris Kennaway <kris@obsecurity.org> To: current@FreeBSD.org, smp@FreeBSD.org, hackers@FreeBSD.org User-Agent: Mutt/1.4.2.2i Cc: Subject: Progress on scaling of FreeBSD on 8 CPU systems Precedence: list Errors-To: owner-freebsd-current@freebsd.org Now that the goals of the SMPng project are complete, for the past year or more several of us have been working hard on profiling FreeBSD in various multiprocessor workloads, and looking for performance bottlenecks to be optimized. We have recently made significant progress on optimizing for MySQL running on an 8-core amd64 system. The graph of results may be found here: http://www.freebsd.org/~kris/scaling/scaling.png This shows the graph of MySQL transactions/second performed by a multi-threaded client workload against a local MySQL database with varying numbers of client threads, with identically configured FreeBSD and Linux systems on the same machine. The test was run on FreeBSD 7.0, with the latest version of the ULE 2.0 scheduler, the libthr threading library, and an uncommitted patch from Jeff Roberson [1] that addresses poor scalability of file descriptor locking (using a new sleepable mutex primitive); this patch is responsible for almost all of the performance and scaling improvements measured. It also includes some other patches (collected in my kris-contention p4 branch) that have been shown to help contention in MySQL workloads in the past (including a UNIX domain socket locking pushdown patch from Robert Watson), but these were shown to only give small individual contributions, with a cumulative effect on the order of 5-10%. With this configuration we are able to achieve performance that is consistent with Linux at peak (the graph shows Linux 2% faster, but this is commensurate with the margin of error coming from variance between runs, so more data is needed to distinguish them), with 8 client threads (=1 thread/CPU core), and significantly outperforms Linux at higher than peak loads, when running on the same hardware. Specifically, beyond 8 client threads FreeBSD has only minor performance degradation (an 8% drop from peak throughput at 8 clients to 20 clients), but Linux collapses immediately above 8 threads, and above 14 threads asymptotes to essentially single-threaded levels. At 20 clients FreeBSD outperforms Linux by a factor of 4. We see this result as part of the payoff we are seeing from the hard work of many developers over the past 7 years. In particular it is a significant validation of the SMP and locking strategies chosen for the FreeBSD kernel in the post-FreeBSD 4.x world. More configuration details and discussion about the benchmark may be found here: http://people.freebsd.org/~kris/scaling/mysql.html Kris ----- End forwarded message ----- -- Jim C. Nasby, Database Architect decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
> From: Kris Kennaway <kris@obsecurity.org> > We have recently made significant progress on optimizing for MySQL > running on an 8-core amd64 system. The graph of results may be found > here: > > http://www.freebsd.org/~kris/scaling/scaling.png > > This shows the graph of MySQL transactions/second performed by a > multi-threaded client workload against a local MySQL database with > varying numbers of client threads, with identically configured FreeBSD > and Linux systems on the same machine. Interesting -- the MySQL/Linux graph is very similar to the graphs from the .nl magazine posted last year. I think this suggests that the "MySQL deficiency" was rather a performance bug in Linux, not in MySQL itself ... -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Arjen van der Meijden
Дата:
Alvaro Herrera wrote: > Interesting -- the MySQL/Linux graph is very similar to the graphs from > the .nl magazine posted last year. I think this suggests that the > "MySQL deficiency" was rather a performance bug in Linux, not in MySQL > itself ... The latest benchmark we did was both with Solaris and Linux on the same box, both showed such a drop. So I doubt its "not in MySQL", although it might be possible to fix the load MySQL's usage pattern poses on a system, via the OS. And since MySQL 5.0.32 is less bad than 4.1.22 on that system. We didn't have time to test 5.0.25 again, but .32 scaled better, so at least some of the scaling issues where actually fixed in MySQL itself. Best regards, Arjen
On Fri, 2007-03-02 at 09:01, Alvaro Herrera wrote: > > From: Kris Kennaway <kris@obsecurity.org> > > > We have recently made significant progress on optimizing for MySQL > > running on an 8-core amd64 system. The graph of results may be found > > here: > > > > http://www.freebsd.org/~kris/scaling/scaling.png > > > > This shows the graph of MySQL transactions/second performed by a > > multi-threaded client workload against a local MySQL database with > > varying numbers of client threads, with identically configured FreeBSD > > and Linux systems on the same machine. > > Interesting -- the MySQL/Linux graph is very similar to the graphs from > the .nl magazine posted last year. I think this suggests that the > "MySQL deficiency" was rather a performance bug in Linux, not in MySQL > itself ... I rather think it's a combination of how MySQL does things and Linux not being optimized to handle that situation. It may well be that the fixes to BSD have simply moved the point at which performance dives off quickly from 50 connections to 300 or something. I'd really like to see freebsd tested on the 32 thread Sun CPU that had such horrible performance with linux, and with many more threads to see if there's still a cliff there somewhere, and to see where postgresql's cliff would be as well. After all, the most interesting part of performance graphs are the ones you see when the system is heading into overload.
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Arjen van der Meijden
Дата:
And here is that latest benchmark we did, using a 8 dual core opteron Sun Fire x4600. Unfortunately PostgreSQL seems to have some difficulties scaling over 8 cores, but not as bad as MySQL. http://tweakers.net/reviews/674 Best regards, Arjen Arjen van der Meijden wrote: > Alvaro Herrera wrote: >> Interesting -- the MySQL/Linux graph is very similar to the graphs from >> the .nl magazine posted last year. I think this suggests that the >> "MySQL deficiency" was rather a performance bug in Linux, not in MySQL >> itself ... > > The latest benchmark we did was both with Solaris and Linux on the same > box, both showed such a drop. So I doubt its "not in MySQL", although it > might be possible to fix the load MySQL's usage pattern poses on a > system, via the OS. And since MySQL 5.0.32 is less bad than 4.1.22 on > that system. We didn't have time to test 5.0.25 again, but .32 scaled > better, so at least some of the scaling issues where actually fixed in > MySQL itself. > > Best regards, > > Arjen > > ---------------------------(end of broadcast)--------------------------- > TIP 7: You can help support the PostgreSQL project by donating at > > http://www.postgresql.org/about/donate >
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Stefan Kaltenbrunner
Дата:
Arjen van der Meijden wrote: > And here is that latest benchmark we did, using a 8 dual core opteron > Sun Fire x4600. Unfortunately PostgreSQL seems to have some difficulties > scaling over 8 cores, but not as bad as MySQL. > > http://tweakers.net/reviews/674 ouch - do I read that right that even after tom's fixes for the "regressions" in 8.2.0 we are still 30% slower then the -HEAD checkout from the middle of the 8.2 development cycle ? Stefan
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Arjen van der Meijden
Дата:
Stefan Kaltenbrunner wrote: > ouch - do I read that right that even after tom's fixes for the > "regressions" in 8.2.0 we are still 30% slower then the -HEAD checkout > from the middle of the 8.2 development cycle ? Yes, and although I tested about 17 different cvs-checkouts, Tom and I weren't really able to figure out where "it" happened. So its a bit of a mystery why the performance is so much worse. Best regards, Arjen
Arjen van der Meijden wrote: > And here is that latest benchmark we did, using a 8 dual core opteron > Sun Fire x4600. Unfortunately PostgreSQL seems to have some difficulties > scaling over 8 cores, but not as bad as MySQL. > > http://tweakers.net/reviews/674 Hmm - interesting reading as always Arjen. Thanks for the notice on this. -- Richard Huxton Archonet Ltd
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Stefan Kaltenbrunner
Дата:
Arjen van der Meijden wrote: > Stefan Kaltenbrunner wrote: >> ouch - do I read that right that even after tom's fixes for the >> "regressions" in 8.2.0 we are still 30% slower then the -HEAD checkout >> from the middle of the 8.2 development cycle ? > > Yes, and although I tested about 17 different cvs-checkouts, Tom and I > weren't really able to figure out where "it" happened. So its a bit of a > mystery why the performance is so much worse. double ouch - losing that much in performance without an idea WHY it happened is really unfortunate :-( Stefan
Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: > Arjen van der Meijden wrote: >> Stefan Kaltenbrunner wrote: >>> ouch - do I read that right that even after tom's fixes for the >>> "regressions" in 8.2.0 we are still 30% slower then the -HEAD checkout >>> from the middle of the 8.2 development cycle ? >> >> Yes, and although I tested about 17 different cvs-checkouts, Tom and I >> weren't really able to figure out where "it" happened. So its a bit of a >> mystery why the performance is so much worse. > double ouch - losing that much in performance without an idea WHY it > happened is really unfortunate :-( Keep in mind that Arjen's test exercises some rather narrow scenarios; IIRC its performance is mostly determined by some complicated bitmap-indexscan cases. So that "30% slower" bit certainly doesn't represent an across-the-board figure. As best I can tell, the decisions the planner happened to be making in late June were peculiarly nicely suited to his test, but not so much for other cases. regards, tom lane
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Arjen van der Meijden
Дата:
On 5-3-2007 21:38 Tom Lane wrote: > Keep in mind that Arjen's test exercises some rather narrow scenarios; > IIRC its performance is mostly determined by some complicated > bitmap-indexscan cases. So that "30% slower" bit certainly doesn't > represent an across-the-board figure. As best I can tell, the decisions > the planner happened to be making in late June were peculiarly nicely > suited to his test, but not so much for other cases. True, its not written as a database-comparison-test, but as a platform-comparison test. As I showed you back then, there where indeed querytypes faster on the final version (I still have that database of executed queries on dev and 8.2 rc1), especially after your three patches. Still, its a pitty that both the general performance and scalability seem to be worse on these platforms. Best regards, Arjen
Re: [kris@obsecurity.org: Progress on scaling of FreeBSD on 8 CPU systems]
От
Stefan Kaltenbrunner
Дата:
Tom Lane wrote: > Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: >> Arjen van der Meijden wrote: >>> Stefan Kaltenbrunner wrote: >>>> ouch - do I read that right that even after tom's fixes for the >>>> "regressions" in 8.2.0 we are still 30% slower then the -HEAD checkout >>>> from the middle of the 8.2 development cycle ? >>> Yes, and although I tested about 17 different cvs-checkouts, Tom and I >>> weren't really able to figure out where "it" happened. So its a bit of a >>> mystery why the performance is so much worse. > >> double ouch - losing that much in performance without an idea WHY it >> happened is really unfortunate :-( > > Keep in mind that Arjen's test exercises some rather narrow scenarios; > IIRC its performance is mostly determined by some complicated > bitmap-indexscan cases. So that "30% slower" bit certainly doesn't > represent an across-the-board figure. As best I can tell, the decisions > the planner happened to be making in late June were peculiarly nicely > suited to his test, but not so much for other cases. understood - I was not trying to imply that we suffer a 30% performance drop overall. But still it means we know about a set of queries that we once could handle faster than we can now ... Stefan