Re: new tests post-feature freeze (was pgsql: Add TAP tests for pg_dump)

On Sun, May 08, 2016 at 12:29:27PM -0400, Stephen Frost wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
> > My suggestion is that, from this point forward, we add new tests to
> > 9.6 only if they are closely related to a bug that is getting fixed or
> > a feature that is new in 9.6.  I think that's a reasonable compromise,
> > but what do others think?

+1.  This is a natural extension of the well-established default that we
(back-)patch tests for a bug into all releases getting a fix for the bug.

> I'm willing to accept that compromise, but I'm not thrilled with it due
> to what it will mean for the process I'm currently going through.  The
> approach I've been using has been to add tests to gain more code
> coverage of the code in pg_dump.  That has turned up multiple
> pre-existing bugs in pg_dump but the vast majority of the tests come
> back clean.  This compromise would mean that I'd continue to work
> through the code coverage tests, but would have to segregate out and
> commit only those tests which actually reveal bugs, once those bugs have
> been fixed (as to avoid turning the buildfarm red).  The rest of the
> tests would still get written, but since they currently don't reveal
> bugs, they would be shelved until development is opened for 9.7.

Some or even most of the other tests would qualify under "closely related to
... a feature that is new in 9.6".  Your 9.6 pg_dump changes affected object
selection and catalog extraction for most object types, so I think validating
those paths is in scope under Robert's suggestion.  Testing "pg_dump
--encoding" or "pg_dump --jobs" probably wouldn't fall in scope, because those
features operate at arm's length from the 9.6 pg_dump changes.  Expanding, for
example, tests of postgres_fdw query deparse would certainly fall out of
scope.  That would have no apparent chance of catching a regression caused by
the 9.6 pg_dump changes.