Обсуждение: Daily regression testing via vmware - useful?

What is the current state of regression testing on the CVS tree?  Are
the regression tests performed once in a while, or routinely?  If
performed once in a while, perhaps the following would idea would be of
interest:

I am kicking around the idea of using one of my old machines (a P90) to
run nightly regression tests on the CVS tree.  The idea would be to use
vmware to set up multiple virtual machines, one for each OS or
distribution (Linux clib, Linux glib, FreeBSD,  etc. etc.).  Perhaps
even multiple versions of the same OS/dist (RH6.0, RH6.1, SuSE 6.1, SuSE
6.2) to catch subtle changes between versions if they are known to be a
problem.

That way, I could conduct a daily automatic regression test on them all.

A program (probably python) would combine the outputs into an HTML table
of OS version vs. test result.  All developers would then be able to
keep an eye on the effects their changes have on other platforms.

Would this be of value , or waste of effort ????

Steve


PS

Looks like I am going to be offline for about 3 weeks so will not be
able to pick up replies to this question until I get back.  If you
reply, be sure to Cc sbirch@ironmountainsystems.com so I don't lose the
reply.

Stephen Birch <sbirch@ironmountainsystems.com> writes:
> What is the current state of regression testing on the CVS tree?  Are
> the regression tests performed once in a while, or routinely?

I make a practice of running them whenever I pull a cvs update (once or
twice a week usually), and before I commit any changes.  I hope other
developers run them fairly routinely as well.

> I am kicking around the idea of using one of my old machines (a P90) to
> run nightly regression tests on the CVS tree.

I think this might be a good thing to do, even though as MikeA points
out, coverage of only Intel-based platforms isn't all that impressive.
(At least not to those of us who use non-Intel platforms.)  If the
idea seems to work out, perhaps other people could be persuaded to
contribute cycles on non-Intel boxes.

There are other open-source projects running automated regression tests
this way; Mozilla is probably the most visible example.  As far as I've
heard, it's been useful for them.  If you can set it up without too much
work, I'd say give it a try, and we'll find out whether it helps us or
not.  We can always drop the experiment if it seems not to help.
        regards, tom lane