Обсуждение: meson and check-tests
Hi Tristan,
--
Using make I can run only selected tests under src/test/regress using TESTS=... make check-tests. I didn't find any way to do that with meson. meson test --suite regress runs all the regression tests.
We talked this off-list at the conference. It seems we have to somehow avoid passing pg_regress --schedule argument and instead pass the list of tests. Any idea how to do that?
Best Wishes,
Ashutosh Bapat
On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: > Hi Tristan, > Using make I can run only selected tests under src/test/regress using > TESTS=... make check-tests. I didn't find any way to do that with meson. > meson test --suite regress runs all the regression tests. > > We talked this off-list at the conference. It seems we have to somehow > avoid passing pg_regress --schedule argument and instead pass the list of > tests. Any idea how to do that? I think there are 2 solutions to this. 1. Avoid passing --schedule by default, which doesn't sound like a great solution. 2. Teach pg_regress to ignore the --schedule option if specific tests are passed instead. 3. Add a --no-schedule option to pg_regress which would override the previously added --schedule option. I personally prefer 2 or 3. 2: meson test -C build regress/regress --test-args my_specific_test 3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test" Does anyone have an opinion? -- Tristan Partin https://tristan.partin.io
On Sun, Jun 2, 2024 at 4:48 AM Tristan Partin <tristan@partin.io> wrote: > > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: > > Hi Tristan, > > Using make I can run only selected tests under src/test/regress using > > TESTS=... make check-tests. I didn't find any way to do that with meson. > > meson test --suite regress runs all the regression tests. > > > > We talked this off-list at the conference. It seems we have to somehow > > avoid passing pg_regress --schedule argument and instead pass the list of > > tests. Any idea how to do that? > > I think there are 2 solutions to this. > > 1. Avoid passing --schedule by default, which doesn't sound like a great > solution. > > 2. Teach pg_regress to ignore the --schedule option if specific tests > are passed instead. > > 3. Add a --no-schedule option to pg_regress which would override the > previously added --schedule option. > > I personally prefer 2 or 3. > > 2: meson test -C build regress/regress --test-args my_specific_test > 3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test" > > Does anyone have an opinion? > if each individual sql file can be tested separately, will `test: test_setup`? be aslo executed as a prerequisite? in # ---------- # The first group of parallel tests # ---------- test: boolean char name varchar text int2 int4 int8 oid float4 float8 bit numeric txid uuid enum money rangetypes pg_lsn regproc # ---------- # The second group of parallel tests # multirangetypes depends on rangetypes # multirangetypes shouldn't run concurrently with type_sanity # ---------- test: strings md5 numerology point lseg line box path polygon circle date time timetz timestamp timestamptz interval inet macaddr macaddr8 multirangetypes If we can test each individual sql file via meson, that would be great. but based on the above comments, we need to preserve the specified order. like: TEST=rangetypes, multirangetypes will first run rangetypes then multirangetypes. can we tag/name each src/test/regress/parallel_schedule groups like: group1:test: boolean char name varchar text int2 int4 int8 oid float4 float8 bit numeric txid uuid enum money rangetypes pg_lsn regproc group2:test: strings md5 numerology point lseg line box path polygon circle date time timetz timestamp timestamptz interval inet macaddr macaddr8 multirangetypes Then, we can test each individual group. TEST=group1, group2.
"Tristan Partin" <tristan@partin.io> writes: > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: >> We talked this off-list at the conference. It seems we have to somehow >> avoid passing pg_regress --schedule argument and instead pass the list of >> tests. Any idea how to do that? > I think there are 2 solutions to this. > 1. Avoid passing --schedule by default, which doesn't sound like a great > solution. > 2. Teach pg_regress to ignore the --schedule option if specific tests > are passed instead. > 3. Add a --no-schedule option to pg_regress which would override the > previously added --schedule option. > I personally prefer 2 or 3. Just to refresh peoples' memory of what the Makefiles do: src/test/regress/GNUmakefile has check: all $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS) check-tests: all | temp-install $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS) (and parallel cases for installcheck etc). AFAICS, meson.build has no equivalent to the EXTRA_TESTS add-on, nor does it have behavior equivalent to check-tests' substitution of $(TESTS) for --schedule. But I suggest that those behaviors have stood for a long time and so the appropriate thing to do is duplicate them as best we can, not invent something different. regards, tom lane
On 2024-06-02 Su 01:25, Tom Lane wrote: > "Tristan Partin" <tristan@partin.io> writes: >> On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: >>> We talked this off-list at the conference. It seems we have to somehow >>> avoid passing pg_regress --schedule argument and instead pass the list of >>> tests. Any idea how to do that? >> I think there are 2 solutions to this. >> 1. Avoid passing --schedule by default, which doesn't sound like a great >> solution. >> 2. Teach pg_regress to ignore the --schedule option if specific tests >> are passed instead. >> 3. Add a --no-schedule option to pg_regress which would override the >> previously added --schedule option. >> I personally prefer 2 or 3. > Just to refresh peoples' memory of what the Makefiles do: > src/test/regress/GNUmakefile has > > check: all > $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS) > > check-tests: all | temp-install > $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS) > > (and parallel cases for installcheck etc). AFAICS, meson.build has > no equivalent to the EXTRA_TESTS add-on, nor does it have behavior > equivalent to check-tests' substitution of $(TESTS) for --schedule. > But I suggest that those behaviors have stood for a long time and > so the appropriate thing to do is duplicate them as best we can, > not invent something different. > > +1 cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On Sun, 2 Jun 2024 at 07:06, jian he <jian.universality@gmail.com> wrote: > > On Sun, Jun 2, 2024 at 4:48 AM Tristan Partin <tristan@partin.io> wrote: > > > > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: > > > Hi Tristan, > > > Using make I can run only selected tests under src/test/regress using > > > TESTS=... make check-tests. I didn't find any way to do that with meson. > > > meson test --suite regress runs all the regression tests. > > > > > > We talked this off-list at the conference. It seems we have to somehow > > > avoid passing pg_regress --schedule argument and instead pass the list of > > > tests. Any idea how to do that? > > > > I think there are 2 solutions to this. > > > > 1. Avoid passing --schedule by default, which doesn't sound like a great > > solution. > > > > 2. Teach pg_regress to ignore the --schedule option if specific tests > > are passed instead. > > > > 3. Add a --no-schedule option to pg_regress which would override the > > previously added --schedule option. > > > > I personally prefer 2 or 3. > > > > 2: meson test -C build regress/regress --test-args my_specific_test > > 3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test" > > > > Does anyone have an opinion? > > > > if each individual sql file can be tested separately, will > `test: test_setup`? > be aslo executed as a prerequisite? Yes, it is required to run at least two setup tests because regress tests require a database to be created: 1- The initdb executable is needed, and it is installed by the 'tmp_install' test for the tests. 2- Although the initdb executable exists, it is not enough by itself. Regress tests copies its database contents from the template directory, instead of running initdb for each test [1] (This is the default behaviour in the meson builds' regress tests). This template directory is created by the 'initdb_cache' test. [1] 252dcb3239 -- Regards, Nazir Bilal Yavuz Microsoft
On Sun Jun 2, 2024 at 12:25 AM CDT, Tom Lane wrote: > "Tristan Partin" <tristan@partin.io> writes: > > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote: > >> We talked this off-list at the conference. It seems we have to somehow > >> avoid passing pg_regress --schedule argument and instead pass the list of > >> tests. Any idea how to do that? > > > I think there are 2 solutions to this. > > 1. Avoid passing --schedule by default, which doesn't sound like a great > > solution. > > 2. Teach pg_regress to ignore the --schedule option if specific tests > > are passed instead. > > 3. Add a --no-schedule option to pg_regress which would override the > > previously added --schedule option. > > I personally prefer 2 or 3. > > Just to refresh peoples' memory of what the Makefiles do: > src/test/regress/GNUmakefile has > > check: all > $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS) > > check-tests: all | temp-install > $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS) > > (and parallel cases for installcheck etc). AFAICS, meson.build has > no equivalent to the EXTRA_TESTS add-on, nor does it have behavior > equivalent to check-tests' substitution of $(TESTS) for --schedule. > But I suggest that those behaviors have stood for a long time and > so the appropriate thing to do is duplicate them as best we can, > not invent something different. In theory, this makes sense. In practice, this is hard to emulate. We could make the check-tests a Meson run_target() instead of another test(), which would end up running the same tests more than once. We could also add the same functionality proposed in my email to the Makefile, so they mirror each other. check-tests would then just become a legacy target name. -- Tristan Partin https://tristan.partin.io
On Mon, Jun 3, 2024 at 10:04 PM Tristan Partin <tristan@partin.io> wrote:
On Sun Jun 2, 2024 at 12:25 AM CDT, Tom Lane wrote:
> "Tristan Partin" <tristan@partin.io> writes:
> > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
> >> We talked this off-list at the conference. It seems we have to somehow
> >> avoid passing pg_regress --schedule argument and instead pass the list of
> >> tests. Any idea how to do that?
>
> > I think there are 2 solutions to this.
> > 1. Avoid passing --schedule by default, which doesn't sound like a great
> > solution.
> > 2. Teach pg_regress to ignore the --schedule option if specific tests
> > are passed instead.
> > 3. Add a --no-schedule option to pg_regress which would override the
> > previously added --schedule option.
> > I personally prefer 2 or 3.
>
> Just to refresh peoples' memory of what the Makefiles do:
> src/test/regress/GNUmakefile has
>
> check: all
> $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS)
>
> check-tests: all | temp-install
> $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS)
>
> (and parallel cases for installcheck etc). AFAICS, meson.build has
> no equivalent to the EXTRA_TESTS add-on, nor does it have behavior
> equivalent to check-tests' substitution of $(TESTS) for --schedule.
> But I suggest that those behaviors have stood for a long time and
> so the appropriate thing to do is duplicate them as best we can,
> not invent something different.
In theory, this makes sense. In practice, this is hard to emulate. We
could make the check-tests a Meson run_target() instead of another
test(), which would end up running the same tests more than once.
meson has changed the way we run individual perl tests and that looks better. So I am fine if meson provides a better way to do what `make check-tests` does. But changing pg_regress seems a wrong choice or even making changes to the current make system. Instead we should make meson pass the right arguments to pg_regress. In this case, it should not pass --schedule when we need `make check-tests` like functionality.
Just adding check-tests as new target won't help we need some way to specify "which tests" to run. Thus by default this target should not run any tests? I don't understand meson well. So I might be completely wrong?
How about the following options?
1. TESTS="..." meson test --suite regress - would run the specified tests from regress
2. Make `meson test --suite regress / regress/partition_join` run partition_join.sql test. I am not how to specify multiple tests in this command. May be `meson test --suite regress / regress/test_setup,partition_join` will do that. make check-tests allows one to run multiple tests like TESTS="test_setup partition_join" make check-tests.
Best Wishes,
Ashutosh Bapat