Re: Two weeks to feature freeze

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Friday, June 20, 2003 11:47 PM
> To: Dann Corbit
> Cc: Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
>
>
> "Dann Corbit" <DCorbit@connx.com> writes:
> > Look at this:
> >
> http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on
>
> This looks a little cleaner than the last time I looked at it
> (more than three years ago), but it's still fundamentally a
> marketing effort.  It is not an exercise in spec compliance
> measurement, because there are hundreds of bullet points that
> all look exactly alike, whether they are measuring
> spec-required elements, random vendor extensions to the spec,
> or spec violations.  To take just one example of the latter,
> "Calculate 1--1" is still shown with a green star for MySQL
> and a failure for Postgres, when a more correct reading would
> be "Fails to recognize SQL-standard -- comment syntax" for
> MySQL.  And yes, they were called out on this three years
> ago, and no they haven't fixed the entry since then.  I
> should believe that there is any good faith on their part?
>
> For another example, take a close look at the "Quoting"
> section, which purports to measure compliance to the spec's
> ideas about how to quote an identifier.  Postgres accepts
> double-quoted identifiers per spec, including doubled double
> quotes per spec, and rejects bracketed or backquoted
> identifiers per spec.  MySQL is apparently spec compliant on
> just one of those four points.  Curious that they manage to
> end up with a better looking display than us in this section;
> in particular note that Postgres is specifically claimed
> *not* to handle double-quoted identifiers.  (Memory is fuzzy
> after three years, but IIRC when you look at the actual test
> code being used, it tests more than whether double quoted
> identifiers are allowed, and really is failing us on some
> quite unrelated detail.)
>
> Another point worth mentioning is that most of the numerical
> limits shown in the table have nothing to do with actual
> server limits, but with random limitations of their test
> process.  For instance, I'm not sure what "max index part
> length 235328" really means, but I am pretty sure it's got
> nothing to do with the Postgres server.  Or look at "constant
> string size in SELECT 16777207" ... nope, there's no such
> limit.  (If they'd put a "+" in there then it'd be okay, but
> no.) I still remember watching crash-me trying to measure the
> max query length of Postgres 7.0: the crashme client process
> dumped core before Postgres did, after which the controlling
> script announced that we weren't crash-safe.
>
> > So far, I have seen three problems pointed out (out of 600+ tests).
>
> These are the high spots from three-year-old memories.  Do
> you really want a detailed analysis?  A quick look at their
> table recalls plenty of bogosity to my mind.
>
> A last point is that this table is comparing MySQL 4.1
> (bleeding edge alpha release) against PG 7.2 (one full major
> release behind the times). While I cannot really blame the
> MySQL guys for not being up-to-the- minute on everyone else's
> releases, this does emphasize the key point, namely that this
> isn't a fair comparison run by disinterested parties but a
> marketing effort of, by, and for MySQL.

It seems pretty clear that there are warts on the Crashme test.
Perhaps 70% or so is truly useful.  Maybe the useful subset could be
approximated or modified to be useful as a general tool set.

Not too surprising that a commercial enterprise tries to bend the facts
in their favor a bit.

Some other stuff worth note:
http://osdb.sourceforge.net/
http://sourceforge.net/projects/osdldbt (looks like someone has put a
bunch of PostgreSQL effort into it.
http://sourceforge.net/projects/ltp/ (DOTS)
http://www.mysql.com/portal/software/item-222.html (I won't mention
where it's from)
ftp://ftp.cs.wisc.edu/OO7/


Win32 specific, but has source code:
http://www.mipt.sw.ru/en/install/ots/ (ODBC testing)
http://www.mipt.sw.ru/en/install/ats/ (ADO testing)
Some other interesting stuff is found there too...

Test tools links:
http://www.softwareqatest.com/qattls1.html
http://www.aptest.com/resources.html