Re: New regression test time

On 07/02/2013 10:17 AM, Robert Haas wrote:
> Reviewing this thread, I think that the following people are in favor
> of adding the tests to the existing schedule: Josh Berkus, Stephen
> Frost, Fabien Coelho, Dann Corbit, and Jeff Janes.  And I think that
> the following people are in favor of a new schedule: Andres Freund,
> Andrew Dunstan, David Fetter, and possibly Alvaro Herrera.  I believe
> Tom Lane has also expressed objections, though not on this thread.
>
> So I think the first question we need to answer is: Should we just
> reject Robins' patches en masse?  If we do that, then the rest of this
> is moot.  If we don't do that, then the second question is whether we
> should try to introduce a new schedule, and if so, whether we should
> split out that new schedule before or after committing these patches
> as they stand.
>
> Here are my opinions, for what they are worth.  First, I think that
> rejecting these new tests is a bad idea, although I've looked them
> over a bit and I think there might be individual things we might want
> to take out.  Second, I think that creating a new schedule is likely
> to cost developers more time than it saves them.  Sure, you'll be able
> to run the tests slightly faster, but when you commit something, break
> the buildfarm (which does run those tests), and then have to go back
> and fix the tests (or your patch), you'll waste more time doing that
> than you saved by avoiding those few extra seconds of runtime.  Third,
> I think if we do want to create a new schedule, it makes more sense to
> commit the tests first and then split out the new schedule according
> to some criteria that we devise.  There should be a principled reason
> for putting tests in one schedule or the other; "all the tests that
> Robins Tharakan wrote" is not a good filter criteria.
>
> I'm willing to put effort into going through these patches and
> figuring out which parts are worth committing, and commit them.
> However, I don't want to (and should not) do that if the consensus is
> to reject the patches altogether; or if people are not OK with the
> approach proposed above, namely, commit it first and then, if it
> causes problems, decide how to fix it.  Please help me understand what
> the way forward is for this patch set.
>


I think I'm probably a bit misrepresented here. The question of what 
tests we have is distinct from the question of what schedule(s) they are 
in. We already have tests that are in NO schedule, IIRC. What is more, 
it's entirely possibly to invoke pg_regress with multiple --schedule 
arguments, so we could, for example, have a makefile target that would 
run both the check and some other schedule of longer running tests.

So my $0.02 says you should assess these tests on their own merits, and 
we can debate the schedule arrangements later.

cheers

andrew