On Sun, May 8, 2016 at 6:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
> I do think that now is a good time for people to be reviewing what's
> been committed, which includes writing tests (either automated ones,
> which can be included in our test suite, or one-off's for testing
> specific things but which don't make sense to include).
And what if the tests are buggy? New test suites should go through a
CF first I think for proper review. And this is clearly one.
> Regarding when we should stop adding new tests, I can't agree with the
> notion that they should be treated as new features. New tests may break
> the buildfarm, but they do not impact the stability of committed code,
> nor do they introduce new bugs into the code which has been committed
> (if anything, they expose committed bugs, as the set of tests we're
> discussing did, which should then be fixed).
Yes, new tests do not impact the core code, but they could hide
existing bugs, which is what I think Peter is arguing about here, and
why it is not a good idea to pushed in a complete new test suite just
before a beta release. The buildfarm is made so as a run stops once of
the tests turns red, not after all of them have run.
> As such, I'd certainly encourage you to propose new tests, even now, but
> not changes to the PG code, except for bug fixes, or changes agreed to
> by the RMT. How you prioritise that with the release preparation work
> is up to you, of course, though if I were in your shoes, I'd take care
> of the release prep first, as we're quite close to doing a set of
> releases. I'm happy to try and help with that effort myself, though
> I've not been involved very much in release prep and am not entirely
> sure what I can do to help.
Adding new tests that are part of an existing bug fix is fine because
the failure of such a test would mean that the bug fix is not working
as intended. Based on my reading of the code this adds test coverage
that is larger than the basic test for a bug fix.
--
Michael