On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > FWIW, I've objected in the past to tests that would significantly > increase the runtime of "make check", unless I thought they were > especially valuable (which enumerating every minor behavior of a > feature patch generally isn't IMO). I still think that that's an > important consideration: every second you add to "make check" is > multiplied many times over when you consider how many developers > run that how many times a day. > > We've talked about having some sort of second rank of tests that > people wouldn't necessarily run before committing, and that would > be allowed to eat more time than the core regression tests would. > I think that might be a valuable direction to pursue if people start > submitting very bulky tests.
Maybe. Adding a whole new test suite is significantly more administratively complex, because the BF client has to get updated to run it. And if expected outputs in that test suite change very often at all, then committers will have to run it before committing anyway.
The value of a core regression suite that takes less time to run has to be weighed against the possibility that a better core regression suite might cause us to find more bugs before committing. That could easily be worth the price in runtime.
Seems like a simple fix. We maintain all regression tests in full, but keep slow tests in separate files accessed only by a different schedule.
make check == fast-parallel_schedule
make check-full == parallel_schedule
Probably easier to make one schedule call the other, so we don't duplicate anything.
Tom gets his fast schedule, others get their full schedule.
--
Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services