Обсуждение: Other benchmark than OSDB
Hi everyone!
I have searched the net looking for other postgreSQL benchmark
than the one offered by OSDB, and couldn'd find anything.
Is there other on the net?
..............................................
A Question...
Since before your sun burned hot in space
and before your race was born,
I have awaited a question.
Elielson Fontanezi
DBA Technical Support - PRODAM
Parque do Ibirapuera s/n - SP - BRAZIL
+55 11 5080 9493
Someone needs to check out http://www.osdl.org/projects/performance/osdldbt.html the OSDL benchmarking tool for DBs and get Postgres to work with it and then run it and publish results or at a mininum use it to help tune performance for Postgres. -Derek Elielson Fontanezi said: > Hi everyone! > > I have searched the net looking for other postgreSQL benchmark > than the one offered by OSDB, and couldn'd find anything. > Is there other on the net? > > .............................................. > A Question... > Since before your sun burned hot in space > and before your race was born, > I have awaited a question. > > Elielson Fontanezi > DBA Technical Support - PRODAM > Parque do Ibirapuera s/n - SP - BRAZIL > +55 11 5080 9493 > >
On Tue, Sep 17, 2002 at 11:53:05AM -0700, Derek Neighbors wrote: > Someone needs to check out > http://www.osdl.org/projects/performance/osdldbt.html the OSDL > benchmarking tool for DBs and get Postgres to work with it and then run > it and publish results or at a mininum use it to help tune performance for > Postgres. I'm afraid that the OSDL benchmark has a (IMNSHO stupid) handicap in getting it working and making it a well-respected benchmark: "While the inspiration for OSDL-DBT-1 is the TPC-W, they are entirely different workloads and results obtained should not and cannot be compared. Commercial use of results obtained by running OSDL-DBT-1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ are expressly prohibited." ^^^^^^^^^^^^^^^^^^^^^^^^ So, it's not TPC, you can't compare it to TPC, and if you wanted to compete with the TPC using this test, you couldn't do it. What's worse, since there's no definition of "commercial use" that I was able to find, I'm not sure if it extends to such things as (for instance) writing case studies or responses to RFPs. Even stranger is that the Sourceforge page says the project is released under the Artistic License, which sure doesn't include restrictions on commercial use. (Indeed, the point of the open source brand is supposedly that it makes free software more appealing to business. Oh, well. :-/ ) I'm sure my boss would _love_ for me to work on something we can't ever use. Sheesh. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Tue, 17 Sep 2002 12:30:25 -0700, Andrew Sullivan wrote: > On Tue, Sep 17, 2002 at 11:53:05AM -0700, Derek Neighbors wrote: >> Someone needs to check out >> http://www.osdl.org/projects/performance/osdldbt.html the OSDL >> benchmarking tool for DBs and get Postgres to work with it and then >> run it and publish results or at a mininum use it to help tune >> performance for Postgres. > > I'm afraid that the OSDL benchmark has a (IMNSHO stupid) handicap in > getting it working and making it a well-respected benchmark: There is no handicap, it just needs to be done. Stored procedures need to be written for postgres. You can't always expect stored procedures for one database to work on another. Similarly, while a lot of databases support ODBC, you can't expect ODBC to be used for every database. We are very willing to support and aid in getting postgres to work with our test kits, but we do not have the resources to commit someone here to do all the work. Please contact me (markw@osdl.org) if anyone wishes to help. > "While the inspiration for OSDL-DBT-1 is the TPC-W, they are entirely > different workloads and results obtained should not and cannot be > compared. Commercial use of results obtained by running OSDL-DBT-1 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > are expressly prohibited." > ^^^^^^^^^^^^^^^^^^^^^^^^ > > So, it's not TPC, you can't compare it to TPC, and if you wanted to > compete with the TPC using this test, you couldn't do it. What's worse, > since there's no definition of "commercial use" that I was able to find, > I'm not sure if it extends to such things as (for instance) writing case > studies or responses to RFPs. Even stranger is that the Sourceforge > page says the project is released under the Artistic License, which sure > doesn't include restrictions on commercial use. (Indeed, the point of > the open source brand is supposedly that it makes free software more > appealing to business. Oh, well. :-/ ) "Commercial use" can be generalized to mean that you cannot use our test kits to promote sales for your product. I hope it is understood that there are legal ramifications regarding comparing results from our test kits to TPC results. I do not follow the jump from restrictions for commercial use to the writing of case studies. For example, there are no issues for using our test kits to write a case study that surveys the performance of various open-source databases. Now you cannot publish performance results for a lot of commercial databases because they have clauses against doing such publications attached to the license of their use. I hope none of the open-source databae are licensed this way. > I'm sure my boss would _love_ for me to work on something we can't ever > use. Sheesh. What exactly do you want to do it for? I can certainly verify if it's possible or not. > A ----- Mark Wong - - markw@osdl.org Open Source Development Lab Inc - A non-profit corporation 15275 SW Koll Parkway - Suite H - Beaverton OR, 97006 (503)-626-2455 x 32 (office) (503)-626-2436 (fax) http://www.osdl.org/archive/markw/
On Thu, Sep 19, 2002 at 12:35:05PM -0700, Mark Wong wrote:
> On Tue, 17 Sep 2002 12:30:25 -0700, Andrew Sullivan wrote:
> > I'm afraid that the OSDL benchmark has a (IMNSHO stupid) handicap in
> > getting it working and making it a well-respected benchmark:
>
> There is no handicap, it just needs to be done.
My point was that the handicap is one having to do with restrictions:
certain kinds of people (like my bosses) are unlikely to fund the
work if they can't use it commercially afterwards.
> "Commercial use" can be generalized to mean that you cannot use our test
> kits to promote sales for your product.
That's still hopelessly vague. I have a piece of software. I use
Postgres (or SAPdb, for that matter). I want to prove that my
free-software-backed product is just as good as anything else. So I
want to publish some benchmarks. According to the above, I can't.
So, the benchmarks may be used, I guess, for internal evaluation
only. I guess a shop that does a lot of that will find a reason to
port these benchmarks to other database engines. But by restricting
the use of the benchmarks for advertising use, the restriction
reduces the pool of potential contributors.
It's not my project, of course, and I don't want to tell anyone else
what to do. I apologise, because re-reading my original message, I
think it reads as though I was saying, "The project should have
such-and-this license." If I want such a project, I can start it
myself.
Mostly, I'm frustrated because while the commercial guys have plenty
of money to go and run 10 gazillion TPC tests, the free guys don't
have that money. I'd like to find a way to overcome that. The OSDB
is nice, but lost of people have reservations about AS3AP. So a
TPC-inspired test would be great.
> I hope it is understood that there are legal ramifications
> regarding comparing results from our test kits to TPC results.
Naturally; but it's an uphill battle trying to supplant the TPC
benchmarks, which really are an industry standard.
> I do not follow the jump from restrictions for commercial use to
> the writing of case studies. For example, there are no issues for
> using our test kits to write a case study that surveys the
> performance of various open-source databases.
Only, it would seem, if you don't then go on to argue, "Our product
is based on [some fast free database], as proven by the test kit. So
buy our product." That seems to be straightforwardly prohibited,
since it is certainly commercial use.
> > I'm sure my boss would _love_ for me to work on something we can't ever
> > use. Sheesh.
>
> What exactly do you want to do it for? I can certainly verify if it's
> possible or not.
It would be very nice to have some solid, well-regarded benchmarks to
put in our materials about why our registry software is fast. I
already know PostgreSQL competes in the field. What I need to do is
convince manager types that PostgreSQL is a good product. OSDB is
one such benchmark. Having more than one would be even better.
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
Hi, I received the mail below, responding to some remarks I made here. I think it's good news. A ----- Forwarded message from "Timothy D. Witham" <wookie@osdl.org> ----- Return-path: <wookie@osdl.org> Envelope-to: andrew@libertyrms.info Subject: Some clarifications on what Mark said From: "Timothy D. Witham" <wookie@osdl.org> To: andrew@libertyrms.info I'm in Helsinki an my access is a little screwed up I going to try and be quick on this. Please feel free to post to the list if you wish as I can't right now. :-( >On Tue, Sep 17, 2002 at 11:53:05AM -0700, Derek Neighbors wrote: ?\>> Someone needs to check out >> http://www.osdl.org/projects/performance/osdldbt.html the OSDL >> benchmarking tool for DBs and get Postgres to work with it and then run >>> it and publish results or at a mininum use it to help tune performance for >> Postgres. > > I'm afraid that the OSDL benchmark has a (IMNSHO stupid) handicap in > getting it working and making it a well-respected benchmark: > > "While the inspiration for OSDL-DBT-1 is the TPC-W, they are entirely > different workloads and results obtained should not and cannot be > compared. Commercial use of results obtained by running OSDL-DBT-1 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > are expressly prohibited." > ^^^^^^^^^^^^^^^^^^^^^^^^ > The above is of course wrong. The license as you note below is Artistic and you can use it how you wish. But you can't use our hardware to generate the numbers for marketing purposes. Mark was a little confused on that item. > So, it's not TPC, you can't compare it to TPC, and if you wanted to > compete with the TPC using this test, you couldn't do it. What's > worse, since there's no definition of "commercial use" that I was > able to find, I'm not sure if it extends to such things as (for > instance) writing case studies or responses to RFPs. Even stranger > is that the Sourceforge page says the project is released under the > Artistic License, which sure doesn't include restrictions on > commercial use. (Indeed, the point of the open source brand is > supposedly that it makes free software more appealing to business. > Oh, well. :-/ ) Let me explain the purpose of the tests. They are to provide an open source workload so that developers can share performance information about this type of commercial workload in an open manner. As you are familiar with TPC rules you can't share that sort of working information in the open. But there is no such restrictions on these two workloads. So if you wanted to share information with the Linux Kernel folks on say, Asynic I/O performance changes you can by posting to the wider lists. > > >I'm sure my boss would _love_ for me to work on something we > can't ever use. Sheesh. > If you are a marketing guy I guess you could use the numbers but you wouldn't have anything to compare to. The real purpose was to create the same type of stresses as a commercial database benchmark so that the engineering issues could be worked in the open. Not to create the next marketing benchmark. Tim -- Timothy D. Witham <wookie@osdl.org> Open Source Development Lab, Inc. ----- End forwarded message ----- -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110