Tom Lane:
> Perhaps the real lesson to be learned is that a little more effort
> should be expended on the regression tests. I have a couple of
> suggestions:
>
> 1. As far as I've seen there is no documentation on how to create
> regression tests. This should be documented and made as easy as
> possible, to encourage people to create tests for missing cases.
Excellent idea. If everyone making a new feature could also make a test case
for it this would help us keep the system stable.
> 2. System variations (roundoff error differences, etc) create spurious
> test complaints that make it hard to interpret the results properly.
> Can anything be done to clean this up?
Hmmm, perhaps we could modify the tests to display results through a function
that rounded to the expected precision eg:
instead of
select floatcol, doublecol from testtab;
use
select display(floatcol, 8), display(doublecol, 16) from testtab;
> 3. The TODO list should maintain a section on missing regression tests;
> any failure that gets by the regression tests should cause an entry
> to get made here. This list would have a side benefit of warning
> developers about areas that are not getting tested, so that they know
> they have to do some hand testing if they change relevant code.
>
> We can start the new TODO section with:
>
> * Check destroydb. (Currently, running regression a second time checks
> this, but a single run in a clean tree won't.)
> * Check copy from stdin/to stdout.
> * Check large-object interface.
>
> What else have people been burnt by lately?
The int2, oidint2, int4, and oidint4 tests (and some others I think) are
currently failing because the text of a couple error messages changed and
the "expected" output was not updated. This kind of thing is pretty annoying
as whoever changed the messages really should have updated the tests as well.
If the current messages are preferred to the old messages, I will fix the test
output to match, although personally, I like the old messages better.
I will argue once again for a clean snapshot that is known to pass regression.
This snapshot could be just a CVS tag, but it is important when starting work
on a complex change to be able to know that any problems you have when you
are done are due to your work, not some pre-existing condition.
-dg
David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"I believe OS/2 is destined to be the most important operating
system, and possibly program, of all time" - Bill Gates, Nov, 1987.