Re: New regression test time

On 2013-06-28 14:46:10 -0700, Josh Berkus wrote:
> 
> > How did you evaluate that coverage increased "greatly"? I am not
> > generally against these tests but I'd be surprised if the overall test
> > coverage improved noticeably by this. Which makes 10% runtime overhead
> > pretty hefty if the goal is to actually achieve a high coverage.
> 
> I was relying on Robins' numbers of coverage:

Those improvements rather likely end up being an improvement a good bit
less than one percent for the whole binary.

> If we someday add so many tests that "make check" takes over a minute on
> a modern laptop, then maybe it'll be worth talking about splitting the
> test suite into "regular" and "extended".  However, it would require 15
> more patch sets the size of Robins' to get there, so I don't see that
> it's worth the trouble any time soon.

Was it actually an assert enabled build that you tested?

We currently can run make check with stuff like CLOBBER_CACHE_ALWAYS
which finds bugs pretty regularly. If we achieve a high coverage we
quite possibly can't anymore, at least not regularly.
So I actually think having two modes makes sense. Then we could also
link stuff like isolationtester automatically into the longer running
mode...

Greetings,

Andres Freund

-- Andres Freund	                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services