Hi,
On 2017-02-15 09:30:39 -0800, Jeff Janes wrote:
> On Tue, Feb 14, 2017 at 9:17 PM, Andres Freund <andres@anarazel.de> wrote:
> > What's your reason to get this fixed?
> >
>
> More testing is better than less testing, and a good way to get less
> testing is requiring the tester to memorize a list of false positives they
> might encounter. I'd like to systematically clone my production system,
> run it through pg_upgrade, and then do installcheck (plus my own
> app-specific) on the result, and I'd like others to do that as well with
> their own production systems.
> I don't really see the cost here.
Because that means we essentially need to make sure that our tests pass
with a combination of about ~20-30 behaviour changing gucs, and ~5
different compilation settings that change output. Either we do that
systematically - which'd be a fair amount of effort - or we're not
getting anywhere, because the setting around the next corner breaks a
bunch of different tests.
Should tests pass with random client_min_messages settings? Random
enable_? Switched default_with_oids? statement_timeout? Could go on a
long while.
Having to think about all GUCs/compile time settings that could
potentially affect a test and manually set everything up so they don't
makes writing tests a lot harder, and we'll fail anyway unless it's
checked in an automated fashion. Unless that testing is done you're not
really benefiting, because you can't rely on test failures meaning much.
If we want to have a list of GUCs that we want tests to pass under, then
the proponents of that at least should bring up a buildfarm animal
afterwards that test that reasonable combinations of those pass and
continue to pass.
Alternatively we could ALTER DATABASE a bunch of settings on the
regression database at the start, but I'm not sure that's nice either,
because it makes ad-hoc tests with unusual settings harder.
Greetings,
Andres Freund