Re: Two weeks to feature freeze

> -----Original Message-----
> From: The Hermit Hacker [mailto:scrappy@postgresql.org]
> Sent: Sunday, June 22, 2003 12:30 PM
> To: Jan Wieck
> Cc: The Hermit Hacker; Dann Corbit; Tom Lane; Jason Earl;
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
>
>
> On Sun, 22 Jun 2003, Jan Wieck wrote:
>
> > The Hermit Hacker wrote:
> > > On Fri, 20 Jun 2003, Dann Corbit wrote:
> > >
> > >> Designing tests is busywork.  Desiging tests is boring.  Nobody
> > >> wants to design tests, let alone interpret the results
> and define
> > >> correct baselines.  But testing is very, very important.
> > >
> > > But we do do testing ... we even design testing (in the
> form of the
> > > regression tests) ... we just don't do testing that you
> personally
> > > approve of ... and, from what I've seen so far, you
> aren't willing
> > > to actually put
> > > *your* time where your mouth is ... design some tests and
> submit them to
> > > us ... if they are valid, they will get used ...
> > >
> > > If you feel that crash-me is a valid starting point,
> start there and
> > > see where it takes you ...
> >
> > Not that fast! I didn't take the time to check but it wouldn't
> > surprise me if MySQL's crash-me is GPL'd and copyright MySQL AB.
> > That's not an optimal point to start PostgreSQL test code
> from, is it?
>
> I didn't say to copy it, but if the format is what Dann feels
> is required to be taken seriously, it does give a starting
> point to work from ...
>
> the thing is, as I think it was Tom that pointed out, the
> crash-me is more a feature tester then anything ... but Dann
> appears to be stuck on it as the 'be all, end all of testing
> suites' ...

No.  I think it covers a broad spectrum of functionality.  It is clear
that there are warts in it, and also that it is slanted in a few
instances to turn "bugs into features."  But I think that a large and
thorough test suite that covers all major areas of functionality will
prove useful.  A test suite that covers just as many features and yet is
aimed at honest evaluation would be a big benefit.

The larger and more complete a functionality test suite is, the better.
If a test suite covers ten times the functionality, it will uncover ten
times as many defects.  I think it is part of the responsibility of a
software vendor to ensure that any released product is as free of
defects as possible (even an open source tool set).  All real software
products larger than a few hundred lines of code have some defects in
them.  If you are going to trust your companies data to a software tool,
you would want it to be as free from defects as is possible to achieve,
without rasing the cost prohibitively.

I think that performance testing is also a good idea.  One of the big
benefits of creating a large performance suite is that you can profile
your code and find out where the effort is needed to get a big speed
increase.

I think that the NIST validation suite will be very valuable.  The
coverage of NIST is pretty good, but that test has warts on it too.  You
will find (for instance) that there is not one single index built by
that test suite.  So the joins are all brute force.  Yetch.

If PostgreSQL can pass all three areas of NIST (SQL, ecpg (the pc
directory), and the mc directory) that would be pretty impressive.