On Mon, Mar 19, 2012 at 1:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> For an example in our own
> problem space look at mysql, whose regression tests take well over an
> hour to run on a fast box. So they must be damn near bug-free right?
> Uh, not so much, and I think the fact that developers can't easily run
> their test suite is not unrelated to that.
The other problem with this approach is that it's hard to keep a huge
test suite 100% clean. Changes inevitably introduce behaviour changes
that cause some of the tests to fail. If the test suite is huge then
it's a lot of work to be continually fixing these tests and you're
always behind. If it's always the case that some tests in this huge
suite are failing then it's extra work whenever you make a change to
dig through the results and determine whether any of the failures are
caused by your changes and represent a real problem. Even if you do
the work it's easy to get it wrong and miss a real failure.
My suggestion would be to go ahead and check in the python or perl
script but not make that the pg_regress tests that are run by mak
check. Cherry pick just a good set of tests that test most of the
tricky bits and check that in to run on make test. I tihnk there's
even precedent for that in one of the other modules that has a make
longcheck or make slowcheck or something like that.
--
greg