Re: Two weeks to feature freeze

> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> Sent: Saturday, June 21, 2003 8:50 PM
> To: Dann Corbit
> Cc: Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
>
>
> Dann Corbit wrote:
> > That is the worst possible test plan.  It totally lacks
> organization
> > and there is no hint to define when the feature set has
> been covered.
> > Ad hoc testing is a useful addition, but it cannot replace all the
> > standard tests that have been used by the industry for decades.
> >
> > If you run literally hundreds of tests designed to ensure that your
> > product conforms to ANSI/ISO standards then the bugs that
> are missed
> > will be few and far between.  Unless you are bad at designing tests.
> >
> > Designing tests is busywork.  Desiging tests is boring.
> Nobody wants
> > to design tests, let alone interpret the results and define correct
> > baselines.  But testing is very, very important.
>
> I remember when I was with Great Bridge they said, "Oh, we
> are going to have a test setup and do all sorts of testing to
> improve PostgreSQL."  I told them I doubted their testing was
> going to shake out many more bugs than our existing testing
> setup, and you know what, I was pretty much right.  Sure,
> they found a few, but it wasn't much.

PostgreSQL is a fairly mature product, having been in existence in one
form or another for many years now.

I expect that most of the bugs that surface will be in areas of new
functionality.

Great Bridge had the right idea though.  Let's suppose that they ran
10,000 tests and turned up only one bug.  That would be just as valuable
(if not more so) than turning up 100 bugs.  A large, carefully designed
test system is *proof* of software quality, or at least of the effort to
determine the quality level.  It is also proof of the responsibility of
the software's originators.

Scenario:
You are going to install a tool that your organization will invest its
future in.

Vendor A: "We think our tool is pretty solid and our end users hardly
ever turn up any bugs."

Vendor B:" We think our tool is pretty solid and our 8500 tests
currently show only 3 defects with the released version, and these are
low impact issues.  To view our current database of issues, log onto web
form <page>."

Which tool would you prefer to install?