Re: Meson far from ready on Windows

Hi,

On 2024-06-21 12:20:49 +0100, Dave Page wrote:
> On Thu, 20 Jun 2024 at 21:58, Andres Freund <andres@anarazel.de> wrote:
> That is precisely what https://github.com/dpage/winpgbuild/ was intended
> for - and it works well for PG <= 16.

If we develop it into that - I'd be happy. I mostly want to be able to do
automated tests on windows with all reasonable dependencies. And occasionally
do some interactive investigation, without a lot of setup time.

One small advantage of something outside of PG is that it's easy to add
additional dependencies when developing additional features. Except of course
all the windows packaging systems seem ... suboptimal.


> I don't think it's unreasonable to not support static linking, but I take
> your point.

Separately from this thread: ISTM that on windows it'd be quite beneficial to
link a few things statically, given how annoying dealing with dlls can be?
There's also the perf issue addressed further down.


> My assumption all along was that Meson would replace autoconf etc. before
> anything happened with MSVC, precisely because that's the type of
> environment all the Postgres devs work in primarily. Instead we seem to
> have taken what I think is a flawed approach of entirely replacing the
> build system on the platform none of the devs use, whilst leaving the new
> system as an experimental option on the platforms it will have had orders
> of magnitude more testing.

The old system was a major bottleneck. For one, there was no way to run all
tests. And even the tests that one could run, would run serially, leading to
exceedingly long tests times. While that could partially be addressed by
having both buildsystems in parallel, the old one would frequently break in a
way that one couldn't reproduce on other systems. And resource wise it wasn't
feasible to test both old and new system for cfbot/CI.


> What makes it worse, is that I don't believe anyone was warned about such a
> drastic change. Packagers were told about the git archive changes to the
> tarball generation, but that's it (and we've said before, we can't expect
> packagers to follow all the activity on -hackers).

I'm sure we could have dealt better with it. There certainly was some lack of
of cohesion because I wasn't able to do drive the src/tools/msvc removal and
Michael took up the slack.

But I also don't think it's really fair to say that there was no heads
up. Several people at EDB participated in the removal and buildfarm
maintainers at EDB were repeatedly pinged, to move their buildfarm animals
over.

And of course the meson stuff came out a year earlier and it wouldn't have
been exactly unreasonable


> Well as I noted, that is the point of my Github repo above. You can just go
> download the binaries from the all_deps action - you can even download the
> config.pl and buildenv.pl that will work with those dependencies (those
> files are artefacts of the postgresql action).

For the purpose of CI we'd really need debug builds of most of the libraries -
there are compat issues when mixing debug/non-debug runtimes (we've hit them
occasionally) and not using the debug runtime hides a lot of issues. Of course
also not optimal for CI / BF usage.



> > - Linking the backend dynamically against lz4, icu, ssl, xml, xslt, zstd,
> > zlib
> >   slows down the tests noticeably (~20%).  So I ended up building those
> >   statically.

> Curious. I wonder if that translates into a general 20% performance hit.
> Presumably it would for anything that looks similar to whatever test/tests
> are affected.

FWIW, dynamic linking has a noticeable overhead on other platforms too. A
non-dependencies-enabled postgres can do about 2x the connections-per-second
than a fully kitted out postgres can (basically due to more memory mapping
metadata being copied).  But on windows the overhead is larger because so much
more happens for every new connections, including loading all dlls from
scratch.

I suspect linking a few libraries statically would be quite worth it on
windows. On other platforms it'd be quite inadvisable to statically link
libraries, due to security updates, but for stuff like the EDB windows
installer dynamic linking doesn't really help with that afaict?



> >   I ran into some issue with using a static libintl. I made it work, but
> > for
> >   now reverted to a dynamic one.
> >
> >
> > - Enabling nls slows down the tests by about 15%, somewhat painful. This is
> >   when statically linking, it's a bit worse when linked dynamically :(.
> >
> 
> That one I can imagine being in psql, so maybe not a big issue for most
> real world use cases.

I think it's both psql and backend.  I think partially it's just due the
additional libraries being linked in everywhere (intl and iconv) and partially
it's the additinal indirection that happens in a bunch more places. We have a
bunch of places where we do gettext lookups but never use the result unless
you use DEBUG3 or such, and that not free. It also triggers additional
filesystem lookups (for the translations) and that's not cheap on windows
either.

> >
> > I haven't yet looked into a) uuid b) tcl.  I think those are the only other
> > missing dependencies.
> >
> 
> We really need to replace ossp-uuid on Windows anyway. It's basically
> abandoned these days. I haven't looked to see if the alternatives work on
> Windows now.

Yea, once we have *something* for ossp-uuid, I think we should remove support
for ossp-uuid. We don't do anyone favors by inducing them to install it.


Greetings,

Andres Freund