Re: [HACKERS] Current sources?

> > Perhaps the real lesson to be learned is that a little more effort
> > should be expended on the regression tests.  I have a couple of
> > suggestions:
> > 1. As far as I've seen there is no documentation on how to create
> >    regression tests.  This should be documented and made as easy as
> >    possible, to encourage people to create tests for missing cases.

Hmm. It ain't hard, but afaik the only people who have pushed on the
regression tests are scrappy and myself. We went for years with no
updates to the regression tests at all, and now have a somewhat stable
set of tests which actually measure many features of a s/w build.

> Excellent idea. If everyone making a new feature could also make a
> test case for it this would help us keep the system stable.

This would seem to be a truism. Any takers??

> > 2. System variations (roundoff error differences, etc) create
> > spurious test complaints that make it hard to interpret the results
> > properly. Can anything be done to clean this up?
> Hmmm, perhaps we could modify the tests to display results through a
> function that rounded to the expected precision eg:
>    select display(floatcol, 8), display(doublecol, 16) from testtab;

Gee, maybe we should go back to the original v4.2/v1.0.x behavior of
rounding _all_ double precision floating point results to 6 digits :(

We've worked hard to get all of the regression tests to match at least
one platform (at the moment, Linux/i686) and scrappy has extended the
test mechanism to allow for platform-specific differences. But we don't
have access to all of the supported platforms, so others will need to
help (and they have been, at least some).

> > 3. The TODO list should maintain a section on missing regression
> > tests; any failure that gets by the regression tests should cause an
> > entry to get made here. This list would have a side benefit of
> > warning developers about areas that are not getting tested, so that
> > they know they have to do some hand testing if they change relevant
> > code.

imho it will take more effort to maintain a todo list than to just
submit a patch for testing. I would be happy to maintain the "expected"
output if people would suggest new tests (and better, submit patches for
the test).

> I will argue once again for a clean snapshot that is known to pass
> regression.

I snapshot the system all the time, and then do development for a week
or two or more on that revlocked snapshot. afaik the failures in
regression testing at the time I submitted my last "typing" patches were
due to differences in type conversion behavior; I didn't want the
changed behavior to become formalized until others had had a chance to
test and comment. (btw, no one has, and anyway I'm changing the results
for most of those back to what it was before).

It's pretty clear that many patches are submitted without full
regression testing; either way it would be helpful to post a comment
with patches saying how the patches affect the regression tests, or that
no testing was done. I'd like to see another person test patches before
committing to the source tree, but others might like to see where the
patches/changes are heading even before that so I can see arguments both
ways.

As has been suggested by yourself and others, regression test
contributions would be very helpful; so far the discussion amounts to
asking scrappy and myself to do _more_ work on the regression tests. I'd
like to see someone offering specific help at some point :)

Anyway, if Marc or I led this discussion you will probably just get more
ideas similar to what is already there; more brainstorming on the
hackers list from y'all will lead to some good new ideas so I'll shut up
now...

                     - Tom