Re: controlling meson's parallelism (and some whining)

Hi,

On 2023-10-19 13:44:20 -0400, Robert Haas wrote:
> Twice now, I've had 'meson test' fail because it tried to start too
> many copies of the server at the same time. In the server log, I get
> the complaint about needing to raise SHMMNI. This is a macos machine,
> with kern.sysv.shmmni=32.

Hm. Did you not run into simmilar issues with make check-world? I found the
concurrency of that to be even more variable over a run.


But perhaps there's something else wrong here? Perhaps we should deal with
this in Cluster.pm to some degree? Controlling this from the level of meson
(or make/prove for that matter) doesn't really work well, because different
tests start differently many postgres instances.

How many cores does your machine have? I've run the tests in a loop on my m1
mac mini in the past without running into this issue. It has "only" 8 cores
though, whereas I infer, from you mentioning -j8, that you have more cores?


> The obvious fix to this is to just tell 'meson test' how many processes I'd
> like it to run. I thought maybe I could just do 'meson -j8 test' but that
> does not work, because the option is --num-processes and has no short
> version. Even typing -j8 every time would be kind of annoying; typing
> --num-processes 8 every time is ridiculously verbose.

I've also wondered why there's no support for -j, maybe we should open an
issue...


> My next thought was that there ought to be some environmental variable
> that I could set to control this behavior. But I can't find a list of
> environment variables that affect meson behavior anywhere. I guess the
> authors don't believe in environment variable as a control mechanism.

They indeed do not like them - but there is one in this
case: MESON_TESTTHREADS

There's even documentation for it: https://mesonbuild.com/Unit-tests.html#parallelism


> Or, at the risk of sounding a bit testy, maybe their documentation
> just isn't quite up to par. It's not that hard to find lists of
> options for particular subcommands, either from the tool itself or on
> the web site. But unlike git, where you can do something like 'man
> git-checkout' and actually get more information than the command help
> itself provides, there are no man pages for the main subcommands, and
> I can't really find any good documentation on the web site either.
> Knowing that a certain subcommand has a flag called
> --pkgconfig.relocatable or that some other command has a flag called
> --cross-file CROSS_FILE whose argument is, and I quote, a "File
> describing cross compilation environment," is not good enough.

I agree that meson's documentation is of, let's say, varying quality. But
https://mesonbuild.com/Commands.html#test does link to
https://mesonbuild.com/Unit-tests.html which in turn has the bit about
MESON_TESTTHREADS

I do agree that it'd be nice if the online docs were converted to command
specific manpages...

Greetings,

Andres Freund