Обсуждение: [HACKERS] TAP tests - installcheck vs check
AFAICT, unlike the pg_regress checks, which in the installcheck case run against a running instance of postgres, for TAP tests the only difference is that that for the check case a temp install is done, possibly with some extra contrib modules. Is that correct? If is is, why aren't we providing an installcheck target for tests like recover. In at least one case (buildfarmn jacana) installs are quite expensive (2 or 3 minutes) and if they are pointless as seems to be the case here why can't we just avoid them? cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Apr 24, 2017 at 7:29 AM, Andrew Dunstan <andrew.dunstan@2ndquadrant.com> wrote: > > AFAICT, unlike the pg_regress checks, which in the installcheck case run > against a running instance of postgres, for TAP tests the only > difference is that that for the check case a temp install is done, > possibly with some extra contrib modules. Is that correct? If is is, why > aren't we providing an installcheck target for tests like recover. In at > least one case (buildfarmn jacana) installs are quite expensive (2 or 3 > minutes) and if they are pointless as seems to be the case here why > can't we just avoid them? install.pl deploys by default the dll of modules needed for the tests, so no objections. Don't you think the TAP scripts in src/test/perl should be installed as well? I think that this would make sense for consistency with what other Nix platforms do, but there is no real installation of PGXS there. So perhaps they could be deployed in a different path like scripts/perl? -- Michael VMware vCenter Server www.vmware.com
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: > AFAICT, unlike the pg_regress checks, which in the installcheck case run > against a running instance of postgres, for TAP tests the only > difference is that that for the check case a temp install is done, > possibly with some extra contrib modules. Is that correct? If is is, why > aren't we providing an installcheck target for tests like recover. In at > least one case (buildfarmn jacana) installs are quite expensive (2 or 3 > minutes) and if they are pointless as seems to be the case here why > can't we just avoid them? A lot of those test cases involve setting non-default configuration parameters and/or stopping/starting the postmaster. So I can't see how we would run them against a pre-existing live cluster, which is the usual meaning of "make installcheck". I think what you're imagining is skipping redundant builds of the "tmp_install" tree by using an installation tree with a temporary $PGDATA directory. That seems like a fine idea, but we need another word for it. regards, tom lane
On 04/23/2017 10:33 PM, Tom Lane wrote: > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >> AFAICT, unlike the pg_regress checks, which in the installcheck case run >> against a running instance of postgres, for TAP tests the only >> difference is that that for the check case a temp install is done, >> possibly with some extra contrib modules. Is that correct? If is is, why >> aren't we providing an installcheck target for tests like recover. In at >> least one case (buildfarmn jacana) installs are quite expensive (2 or 3 >> minutes) and if they are pointless as seems to be the case here why >> can't we just avoid them? > A lot of those test cases involve setting non-default configuration > parameters and/or stopping/starting the postmaster. So I can't see how > we would run them against a pre-existing live cluster, which is the usual > meaning of "make installcheck". > > I think what you're imagining is skipping redundant builds of the > "tmp_install" tree by using an installation tree with a temporary $PGDATA > directory. That seems like a fine idea, but we need another word for it. > > That's actually the current meaning of installcheck w.r.t. TAP. See Makefile.global.in. It's what we run (mostly) in the buildfarm for the bin tests. I agree entirely that it's confusing as heck. +1 for inventing a new name. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Sun, Apr 23, 2017 at 10:57 PM, Andrew Dunstan <andrew.dunstan@2ndquadrant.com> wrote: > On 04/23/2017 10:33 PM, Tom Lane wrote: >> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >>> AFAICT, unlike the pg_regress checks, which in the installcheck case run >>> against a running instance of postgres, for TAP tests the only >>> difference is that that for the check case a temp install is done, >>> possibly with some extra contrib modules. Is that correct? If is is, why >>> aren't we providing an installcheck target for tests like recover. In at >>> least one case (buildfarmn jacana) installs are quite expensive (2 or 3 >>> minutes) and if they are pointless as seems to be the case here why >>> can't we just avoid them? >> A lot of those test cases involve setting non-default configuration >> parameters and/or stopping/starting the postmaster. So I can't see how >> we would run them against a pre-existing live cluster, which is the usual >> meaning of "make installcheck". >> >> I think what you're imagining is skipping redundant builds of the >> "tmp_install" tree by using an installation tree with a temporary $PGDATA >> directory. That seems like a fine idea, but we need another word for it. > > That's actually the current meaning of installcheck w.r.t. TAP. See > Makefile.global.in. It's what we run (mostly) in the buildfarm for the > bin tests. > > I agree entirely that it's confusing as heck. +1 for inventing a new name. Yeah. I would have expected installcheck to just skip any tests that don't make sense against an already-installed cluster. I would not expect it to run those tests against some cluster other than the installed cluster. That seems super-weird. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 04/25/2017 10:45 AM, Robert Haas wrote: > On Sun, Apr 23, 2017 at 10:57 PM, Andrew Dunstan > <andrew.dunstan@2ndquadrant.com> wrote: >> On 04/23/2017 10:33 PM, Tom Lane wrote: >>> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >>>> AFAICT, unlike the pg_regress checks, which in the installcheck case run >>>> against a running instance of postgres, for TAP tests the only >>>> difference is that that for the check case a temp install is done, >>>> possibly with some extra contrib modules. Is that correct? If is is, why >>>> aren't we providing an installcheck target for tests like recover. In at >>>> least one case (buildfarmn jacana) installs are quite expensive (2 or 3 >>>> minutes) and if they are pointless as seems to be the case here why >>>> can't we just avoid them? >>> A lot of those test cases involve setting non-default configuration >>> parameters and/or stopping/starting the postmaster. So I can't see how >>> we would run them against a pre-existing live cluster, which is the usual >>> meaning of "make installcheck". >>> >>> I think what you're imagining is skipping redundant builds of the >>> "tmp_install" tree by using an installation tree with a temporary $PGDATA >>> directory. That seems like a fine idea, but we need another word for it. >> That's actually the current meaning of installcheck w.r.t. TAP. See >> Makefile.global.in. It's what we run (mostly) in the buildfarm for the >> bin tests. >> >> I agree entirely that it's confusing as heck. +1 for inventing a new name. > Yeah. I would have expected installcheck to just skip any tests that > don't make sense against an already-installed cluster. I would not > expect it to run those tests against some cluster other than the > installed cluster. That seems super-weird. > I'm in the process of moving all the buildfarm tests to use check instead of installcheck, but in such a way that it doesn't constantly generate redundant installs. When I'm done testing that and we have a new buildfarm client release and we get all those buildfarm owners updated (there are 13 animals not controlled by me running TAP tests) I think we should just rename installcheck. for TAP. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: > On 04/25/2017 10:45 AM, Robert Haas wrote: >>> I agree entirely that it's confusing as heck. +1 for inventing a new name. >> Yeah. I would have expected installcheck to just skip any tests that >> don't make sense against an already-installed cluster. I would not >> expect it to run those tests against some cluster other than the >> installed cluster. That seems super-weird. > I'm in the process of moving all the buildfarm tests to use check > instead of installcheck, but in such a way that it doesn't constantly > generate redundant installs. But is that something only of interest to the buildfarm, or should we do something in the Makefile infrastructure to make it more generally available? regards, tom lane
On 04/25/2017 11:13 AM, Tom Lane wrote: > >> I'm in the process of moving all the buildfarm tests to use check >> instead of installcheck, but in such a way that it doesn't constantly >> generate redundant installs. > But is that something only of interest to the buildfarm, or should we > do something in the Makefile infrastructure to make it more generally > available? It already is. If you have a temp_install with all the required pieces do "make NO_TEMP_INSTALL=1 check" The logic I am adding to the buildfarm is essentially to install contrib and test_modules in the temp install directory as part of their install steps and to check that that's been done before using NO_TEMP_INSTALL. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
A couple of thoughts I've been having that relate to this: The traditional meaning of "installcheck" in GNU packages is to test against the installed code, whereas "check" tests before installation. Our concept of testing against a running server obviously does not apply to many kinds of software, so there is no standard name for it. I think as our test suites expand, the concept of testing against a running server is not going to feasible for many new and interesting things. I can see some marginal appeal for what what we're currently doing, but I don't think it's useful to invest much more effort into it to make it work in more cases and situations. What I'm thinking going forward is: make check - as is make installcheck - test against installed code, but start new instances etc. (which you currently can't do for the main test suite) make runningcheck - test against running server, supported for certain tests make check and installcheck would be very similar except make check creates a temp install and sets some path variables. A trick of sorts I saw in Python packages that might also be worth adopting is to make the temp install use symlinks pointing into the source tree. Then as soon as you rebuild, the temp install is up to date (unless you add or remove entire files). Then the temp install is free after the first run. Thoughts? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services