Re: New regression test time

On Tue, Jul 02, 2013 at 10:17:08AM -0400, Robert Haas wrote:
> So I think the first question we need to answer is: Should we just
> reject Robins' patches en masse?  If we do that, then the rest of this
> is moot.  If we don't do that, then the second question is whether we
> should try to introduce a new schedule, and if so, whether we should
> split out that new schedule before or after committing these patches
> as they stand.

It's sad to simply reject meaningful automated tests on the basis of doubt
that they're important enough to belong in every human-in-the-loop test run.
I share the broader vision for automated testing represented by these patches.

> Here are my opinions, for what they are worth.  First, I think that
> rejecting these new tests is a bad idea, although I've looked them
> over a bit and I think there might be individual things we might want
> to take out.  Second, I think that creating a new schedule is likely
> to cost developers more time than it saves them.  Sure, you'll be able
> to run the tests slightly faster, but when you commit something, break
> the buildfarm (which does run those tests), and then have to go back
> and fix the tests (or your patch), you'll waste more time doing that
> than you saved by avoiding those few extra seconds of runtime.  Third,
> I think if we do want to create a new schedule, it makes more sense to
> commit the tests first and then split out the new schedule according
> to some criteria that we devise.  There should be a principled reason
> for putting tests in one schedule or the other; "all the tests that
> Robins Tharakan wrote" is not a good filter criteria.

+1 for that plan.  I don't know whether placing certain tests outside the main
test sequence would indeed cost more time than it saves.  I definitely agree
that if these new tests should appear elsewhere, some of our existing tests
should also move there.