Re: Two weeks to feature freeze

On Fri, Jun 20, 2003 at 10:04:09PM -0700, Dann Corbit wrote:

> > On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote:
> > 
> > Citing Tom Lane:
> > > > I have been through crash-me in some detail, and it left a
> > > > very bad taste in my mouth.  Don't bother holding it up as an 
> > > > example of good practice.
> > > 
> > > Every single test in their list is interesting and useful.
> > 
> > At least on the version I just saw there are several results 
> > with Postgres that are weird (table names > 500 chars?).  
> 
> It does get silly at a point, but I have seen systems with 128
> characters for table names, column names, etc.  Some people seem to like
> it.  Not me.  Too much typing.

I meant that the real limit on 7.2 was much lower than that unless they
twiddled with sources at compile time (there's no configure switch for
that AFAIR).  Maybe 31 or 63 chars, I don't remember.  Do you really
trust the rest of the test seeing that they came up with a clearly wrong
answer in such a simple test?

They can't even "make vacuum run reliably" on 7.1.  See the performance
test.  Maybe they want to test 7.3 with lazy vacuum in place.  Why don't
they do that?  7.1 is already 2 years old.

> > Other things tested are clearly wrong (things that are = 
> > NULL, 
> 
> Sounds like testing for the existence of a bug.
> X = NULL
> X <= NULL
> X >= NULL
> Etc. must always test false, regardless of the contents of X.  Test for
> equality with NULL is a conformance error if NULL == NULL returns true.

You see, you are saying "sounds like they are testing".  What does the
code actually test?  Which is the right behaviour?  Which behaviour
gets the green point, MySQL's or the right one?  There are lots of
things like this; I don't want to waste my time actually reading the
code to see what the correct answer for each test is.

About the 1--1 thing Tom mentioned: be aware that Postgres happily
accepts the correct 1 - -1 expression, but also correctly fails to
"calculate" 1--1.  Which one gets the green point?  Of course it's the
non-compliant one.

Also they don't test things they don't support.  Is there a test for
subselects?  What about concurrency?  Transactional issues?  What about
performance when they have their "transaction support" enabled?


> So far, I have seen three problems pointed out (out of 600+ tests).
> That's 0.5% defects.  Why not just drop the stupid tests, or bend them
> to test for what they ought to be testing.

There's already a mechanism for testing inside Postgres.  Maybe more
tests are needed, but crash-me offers no real value.

I just became aware that the NIST test suite is quite old.  Maybe what's
needed is to expand it to SQL3 to develop a way of measuring the
compliance level.  But the cost of doing that is probably prohibitive.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)