Re: New regression test time

I don't really see a difference in the first two.  If we were sure the uncovered code had no bugs, we wouldn't need to cover it.  At least if you consider unintended behavior changes to be bugs.  I think it would make more sense to split them up by what computers it makes sense to run them on.

Tests that take too much RAM to be run by everyone.

Tests that take too many CPUs (in order to be meaningful) to run by everyone most of the time.

Tests that take too much disk space...

Tests that take too much wall-clock time....

And maybe that tests that take too much wall-clock time specifically under CLOBBER_CACHE_ALWAYS...

Some of these sets would probably be empty currently, because candidates that belong in them were not committed at all since they were not wanted in the default and they there was no other place to add them.

If we are very worried about how long the tests take, we should probably also spend some time trying to make the existing ones faster.  Parallelization does not cut the test time very much (~20% with 8 CPUs), because the tests are poorly packed.  In a parallel group all the tests finish fast except one, and the whole group is then dominated by that one test.  (The main goal of parallelization is probably not to make the test faster, but to make them more realistic from a concurrency perspective, but if there is little actual parallelism, it doesn't achieve that very well, either).  I don't know how much freedom there is to re-order the tests without breaking dependencies, though.  I think prepared_xacts and stats could be usefully run together, as both take a long time sleeping but impose little real load that would interfere with each other.  Perhaps prepared_xacts could be re-written to get what it needs without the long statement_timeouts.  Testing the timeout itself doesn't seem to be the goal.

Cheers,

Jeff