Обсуждение: [RFC] building postgres with meson

Hi,

For the last year or so I've on and off tinkered with $subject.  I think
it's in a state worth sharing now.  First, let's look at a little
comparison.

My workstation:

non-cached configure:
current:        11.80s
meson:           6.67s

non-cached build (world-bin):
current:        40.46s
ninja:           7.31s

no-change build:
current:         1.17s
ninja:           0.06s

test world:
current:          105s
meson:             63s


What actually started to motivate me however were the long times windows
builds took to come back with testsresults. On CI, with the same machine
config:

build:
current:          202s (doesn't include genbki etc)
meson+ninja:      140s
meson+msbuild:    206s


test:
current:         1323s (many commands)
meson:            903s (single command)

(note that the test comparison isn't quite fair - there's a few tests
missing, but it's just small contrib ones afaik)


The biggest difference to me however is not the speed, but how readable
the output is.

Running the tests with meson in a terminal, shows the number of tests
that completed out of how many total, how much time has passed, how long
the currently running tests already have been running.

At the end of a testrun a count of tests is shown:

188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl                         OK
39.51s  110 subtests passed
 
189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation                          OK
62.93s


Ok:                 188
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


The log has the output of the tests and ends with:

Summary of Failures:
120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl                               ERROR             7.16s
(exitstatus 255 or signal 127 SIGinvalid)
 


Quite the difference to make check-world -jnn output.


So, now that the teasing is done, let me explain a bit what lead me down
this path:

Autoconf + make is not being actively developed. Especially autoconf is
*barely* in maintenance mode - despite many shortcomings and bugs. It's
also technology that very few want to use - autoconf m4 is scary, and
it's scarier for people that started more recently than a lot of us
committers for example.

Recursive make as we use it is hard to get right. One reason the clean
make build is so slow compared to meson is that we had to resort to
.NOTPARALLEL to handle dependencies in a bunch of places. And despite
that, I quite regularly see incremental build failures that can be
resolved by retrying the build.

While we have incremental build via --enable-depend, they don't work
that reliable (i.e. misses necessary rebuilds) and yet is often too
aggressive. More modern build system can keep track of the precise
command used to build a target and rebuild it when that command changes.


We also don't just have the autoconf / make buildsystem, there's also
the msvc project generator - something most of us unix-y folks do not
like to touch. I think that, combined with there being no easy way to
run all tests, and it being just different, really hurt our windows
developer appeal (and subsequently the quality of postgres on
windows). I'm not saying this to ding the project generator - that was
well before there were decent "meta" buildsystems out there (and in some
ways it is a small one itself).


The last big issue I have with the current situation is that there's no
good test integration. make check-world output is essentially unreadable
/ not automatically parseable. Which led to the buildfarm having a
separate list of things it needs to test, so that failures can be
pinpointed and paired with appropriate logs. That approach unfortunately
doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
fair bit.


This all led to me to experiment with improvements. I tried a few
somewhat crazy but incremental things like converting our buildsystem to
non-recursive make (I got it to build the backend, but it's too hard to
do manually I think), or to not run tests during the recursive make
check-world, but to append commands to a list of tests, that then is run
by a helper (can kinda be made to work).  In the end I concluded that
the amount of time we'd need to invest to maintain our more-and-more
custom buildsystem going forward doesn't make sense.


Which lead me to look around and analyze which other buildsystems there
are that could make some sense for us. The halfway decent list includes,
I think:
1) cmake
2) bazel
3) meson


cmake would be a decent choice, I think. However, I just can't fully
warm up to it. Something about it just doesn't quite sit right with
me. That's not a good enough reason to prevent others from suggesting to
use it, but it's good enough to justify not investing a lot of time in
it myself.

Bazel has some nice architectural properties. But it requires a JVM to
run - I think that basically makes it insuitable for us. And the build
information seems quite arduous to maintain too.

Which left me with meson. It is a meta-buildsystem that can do the
actual work of building via ninja (the most common one, also targeted by
cmake), msbuild (visual studio project files, important for GUI work)
and xcode projects (I assume that's for a macos IDE, but I haven't tried
to use it). Meson roughly does what autoconf+automake did, in a
python-esque DSL, and outputs build-instructions for ninja / msbuild /
xcode.  One interesting bit is that meson itself is written in python (
and fairly easy to contribute too - I got a few changes in now).


I don't think meson is perfect architecturally - e.g. its insistence on
not having functions ends up making it a bit harder to not end up
duplicating code. There's some user-interface oddities that are now hard
to fix fully, due to the faily wide usage. But all-in-all it's pretty
nice to use.


Its worth calling out that a lot of large open source projects have been
/ are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
linux and also widely used in other platforms), a good chunk of GNOME,
and quite a few more. Due to that it seems unlikely to be abandoned
soon.


As far as I can tell the only OS that postgres currently supports that
meson doesn't support is HPUX. It'd likely be fairly easy to add
gcc-on-hpux support, a chunk more to add support for the proprietary
ones.


The attached patch (meson support is 0016, the rest is prerequisites
that aren't that interesting at this stage) converts most of postgres to
meson. There's a few missing contrib modules, only about half the
optional library dependencies are implemented, and I've only built on
x64. It builds on freebsd, linux, macos and windows (both ninja and
msbuild) and cross builds from linux to windows. Thomas helped make the
freebsd / macos pieces a reality, thanks!

I took a number of shortcuts (although there used to be a *lot*
more). So this shouldn't be reviewed to the normal standard of the
community - it's a prototype.  But I think it's in a complete enough
shape that it allows to do a well-informed evaluation.

What doesn't yet work/ build:

- plenty optional libraries, contrib, NLS, docs build

- PGXS - and I don't yet know what to best do about it. One
  backward-compatible way would be to continue use makefiles for pgxs,
  but do the necessary replacement of Makefile.global.in via meson (and
  not use that for postgres' own build).  But that doesn't really
  provide a nicer path for building postgres extensions on windows, so
  it'd definitely not be a long-term path.

- JIT bitcode generation for anything but src/backend.

- anything but modern-ish x86. That's proably a small amount of work,
  but something that needs to be done.

- exporting all symbols for extension modules on windows (the stuff for
  postgres is implemented). Instead I marked the relevant symbols als
  declspec(dllexport). I think we should do that regardless of the
  buildsystem change. Restricting symbol visibility via gcc's
  -fvisibility=hidden for extensions results in a substantially reduced
  number of exported symbols, and even reduces object size (and I think
  improves the code too).  I'll send an email about that separately.




There's a lot more stuff to talk about, but I'll stop with a small bit
of instructions below:


Demo / instructions:
# Get code
git remote add andres git@github.com:anarazel/postgres.git
git fetch andres
git checkout --track andres/meson

# setup build directory
meson setup build --buildtype debug
cd build

# build (uses automatically as many cores as available)
ninja

# change configuration, build again
meson configure -Dssl=openssl
ninja

# run all tests
meson test

# run just recovery tests
meson test --suite setup --suite recovery

# list tests
meson test --list


Greetings,

Andres Freund

Hi,

On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> non-cached build (world-bin):
> current:        40.46s
> ninja:           7.31s

Interestingly this is pretty close to the minimum achievable on my
machine from the buildsystem perspective.

A build with -fuse-ld=lld, which the above didn't use, takes 6.979s. The
critical path is

bison gram.y -> gram.c       4.13s
gcc gram.c -> gram.o         2.05s
gcc postgres ....            0.317


A very helpful visualization is to transform ninja's build logs into a
tracefile with https://github.com/nico/ninjatracing

I attached an example - the trace.json.gz can be uploaded as-is to
https://ui.perfetto.dev/

It's quite a bit of of fun to look at imo.

There's a few other things quickly apparent:

- genbki prevents build progress due to dependencies on the generated
  headers.
- the absolutely stupid way I implemented the python2->python3
  regression test output conversion uses up a fair bit of resources
- tablecmds.c, pg_dump.c, xlog.c and a few other files are starting to
  big enough to be problematic compile-time wise

Greetings,

Andres Freund

On 12.10.21 10:37, Andres Freund wrote:
> For the last year or so I've on and off tinkered with $subject.  I think
> it's in a state worth sharing now.  First, let's look at a little
> comparison.

I played with $subject a few years ago and liked it.  I think, like you 
said, meson is the best way forward.  I support this project.

One problem I noticed back then was that some choices that we currently 
determine ourselves in configure or the makefiles are hardcoded in 
meson.  For example, at the time, gcc on macOS was not supported.  Meson 
thought, if you are on macOS, you are surely using the Apple compiler, 
and it supports these options.  Fixing that required patches deep in the 
bowels of the meson source code (and, in practice, waiting for a new 
release etc.).  I strongly suspect this isn't the only such problem. 
For example, the shared library build behavior has been carefully tuned 
in opinionated ways.  With the autotools chain, one can override 
anything with enough violence; so we have always felt free to do that. 
I haven't followed it in a while, so I don't know what the situation is 
now; but it is a concern, because we have always felt free to try new 
and unusual build tools (Sun compiler, Intel compiler, 
clang-when-it-was-new) early without waiting for anyone else.

On Tue, Oct 12, 2021 at 9:31 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in
> meson.  For example, at the time, gcc on macOS was not supported.  Meson
> thought, if you are on macOS, you are surely using the Apple compiler,
> and it supports these options.  Fixing that required patches deep in the
> bowels of the meson source code (and, in practice, waiting for a new
> release etc.).  I strongly suspect this isn't the only such problem.
> For example, the shared library build behavior has been carefully tuned
> in opinionated ways.  With the autotools chain, one can override
> anything with enough violence; so we have always felt free to do that.
> I haven't followed it in a while, so I don't know what the situation is
> now; but it is a concern, because we have always felt free to try new
> and unusual build tools (Sun compiler, Intel compiler,
> clang-when-it-was-new) early without waiting for anyone else.

I think we're going to need some solution to this problem. We have too
many people here with strong opinions about questions like this for me
to feel good about the idea that we're going to collectively be OK
with leaving these sorts of decisions up to some other project.

From my point of view, the time it takes to run configure is annoying,
but the build time is pretty fine. On my system, configure takes about
33 seconds, and a full rebuild with 'make -j8' takes 14.5 seconds (I
am using ccache). Moreover, most of the time when I run make, I'm only
doing a partial rebuild, so it's near-instantaneous.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

On 10/12/21 4:37 AM, Andres Freund wrote:
> git remote add andres git@github.com:anarazel/postgres.git

ITYM:

git remote add andres git://github.com/anarazel/postgres.git

cheers

andrew
 
--
Andrew Dunstan
EDB: https://www.enterprisedb.com

.

út 12. 10. 2021 v 10:37 odesílatel Andres Freund <andres@anarazel.de> napsal:
>
> Hi,
>
> For the last year or so I've on and off tinkered with $subject.  I think
> it's in a state worth sharing now.  First, let's look at a little
> comparison.
>
> My workstation:
>
> non-cached configure:
> current:        11.80s
> meson:           6.67s
>
> non-cached build (world-bin):
> current:        40.46s
> ninja:           7.31s
>
> no-change build:
> current:         1.17s
> ninja:           0.06s
>
> test world:
> current:          105s
> meson:             63s
>
>
> What actually started to motivate me however were the long times windows
> builds took to come back with testsresults. On CI, with the same machine
> config:
>
> build:
> current:          202s (doesn't include genbki etc)
> meson+ninja:      140s
> meson+msbuild:    206s
>
>
> test:
> current:         1323s (many commands)
> meson:            903s (single command)
>
> (note that the test comparison isn't quite fair - there's a few tests
> missing, but it's just small contrib ones afaik)
>
>
> The biggest difference to me however is not the speed, but how readable
> the output is.
>
> Running the tests with meson in a terminal, shows the number of tests
> that completed out of how many total, how much time has passed, how long
> the currently running tests already have been running.
>
> At the end of a testrun a count of tests is shown:
>
> 188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl                         OK
39.51s  110 subtests passed 
> 189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation                          OK
62.93s
>
>
> Ok:                 188
> Expected Fail:      0
> Fail:               1
> Unexpected Pass:    0
> Skipped:            0
> Timeout:            0
>
> Full log written to /tmp/meson/meson-logs/testlog.txt
>
>
> The log has the output of the tests and ends with:
>
> Summary of Failures:
> 120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl                               ERROR             7.16s
(exitstatus 255 or signal 127 SIGinvalid) 
>
>
> Quite the difference to make check-world -jnn output.
>
>
> So, now that the teasing is done, let me explain a bit what lead me down
> this path:
>
> Autoconf + make is not being actively developed. Especially autoconf is
> *barely* in maintenance mode - despite many shortcomings and bugs. It's
> also technology that very few want to use - autoconf m4 is scary, and
> it's scarier for people that started more recently than a lot of us
> committers for example.
>
> Recursive make as we use it is hard to get right. One reason the clean
> make build is so slow compared to meson is that we had to resort to
> .NOTPARALLEL to handle dependencies in a bunch of places. And despite
> that, I quite regularly see incremental build failures that can be
> resolved by retrying the build.
>
> While we have incremental build via --enable-depend, they don't work
> that reliable (i.e. misses necessary rebuilds) and yet is often too
> aggressive. More modern build system can keep track of the precise
> command used to build a target and rebuild it when that command changes.
>
>
> We also don't just have the autoconf / make buildsystem, there's also
> the msvc project generator - something most of us unix-y folks do not
> like to touch. I think that, combined with there being no easy way to
> run all tests, and it being just different, really hurt our windows
> developer appeal (and subsequently the quality of postgres on
> windows). I'm not saying this to ding the project generator - that was
> well before there were decent "meta" buildsystems out there (and in some
> ways it is a small one itself).
>
>
> The last big issue I have with the current situation is that there's no
> good test integration. make check-world output is essentially unreadable
> / not automatically parseable. Which led to the buildfarm having a
> separate list of things it needs to test, so that failures can be
> pinpointed and paired with appropriate logs. That approach unfortunately
> doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
> fair bit.
>
>
> This all led to me to experiment with improvements. I tried a few
> somewhat crazy but incremental things like converting our buildsystem to
> non-recursive make (I got it to build the backend, but it's too hard to
> do manually I think), or to not run tests during the recursive make
> check-world, but to append commands to a list of tests, that then is run
> by a helper (can kinda be made to work).  In the end I concluded that
> the amount of time we'd need to invest to maintain our more-and-more
> custom buildsystem going forward doesn't make sense.
>
>
> Which lead me to look around and analyze which other buildsystems there
> are that could make some sense for us. The halfway decent list includes,
> I think:
> 1) cmake
> 2) bazel
> 3) meson
>
>
> cmake would be a decent choice, I think. However, I just can't fully
> warm up to it. Something about it just doesn't quite sit right with
> me. That's not a good enough reason to prevent others from suggesting to
> use it, but it's good enough to justify not investing a lot of time in
> it myself.
>
> Bazel has some nice architectural properties. But it requires a JVM to
> run - I think that basically makes it insuitable for us. And the build
> information seems quite arduous to maintain too.
>
> Which left me with meson. It is a meta-buildsystem that can do the
> actual work of building via ninja (the most common one, also targeted by
> cmake), msbuild (visual studio project files, important for GUI work)
> and xcode projects (I assume that's for a macos IDE, but I haven't tried
> to use it). Meson roughly does what autoconf+automake did, in a
> python-esque DSL, and outputs build-instructions for ninja / msbuild /
> xcode.  One interesting bit is that meson itself is written in python (
> and fairly easy to contribute too - I got a few changes in now).
>
>
> I don't think meson is perfect architecturally - e.g. its insistence on
> not having functions ends up making it a bit harder to not end up
> duplicating code. There's some user-interface oddities that are now hard
> to fix fully, due to the faily wide usage. But all-in-all it's pretty
> nice to use.
>
>
> Its worth calling out that a lot of large open source projects have been
> / are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
> linux and also widely used in other platforms), a good chunk of GNOME,
> and quite a few more. Due to that it seems unlikely to be abandoned
> soon.
>
>
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
>
>
> The attached patch (meson support is 0016, the rest is prerequisites
> that aren't that interesting at this stage) converts most of postgres to
> meson. There's a few missing contrib modules, only about half the
> optional library dependencies are implemented, and I've only built on
> x64. It builds on freebsd, linux, macos and windows (both ninja and
> msbuild) and cross builds from linux to windows. Thomas helped make the
> freebsd / macos pieces a reality, thanks!
>
> I took a number of shortcuts (although there used to be a *lot*
> more). So this shouldn't be reviewed to the normal standard of the
> community - it's a prototype.  But I think it's in a complete enough
> shape that it allows to do a well-informed evaluation.
>
> What doesn't yet work/ build:
>
> - plenty optional libraries, contrib, NLS, docs build
>
> - PGXS - and I don't yet know what to best do about it. One
>   backward-compatible way would be to continue use makefiles for pgxs,
>   but do the necessary replacement of Makefile.global.in via meson (and
>   not use that for postgres' own build).  But that doesn't really
>   provide a nicer path for building postgres extensions on windows, so
>   it'd definitely not be a long-term path.
>
> - JIT bitcode generation for anything but src/backend.
>
> - anything but modern-ish x86. That's proably a small amount of work,
>   but something that needs to be done.
>
> - exporting all symbols for extension modules on windows (the stuff for
>   postgres is implemented). Instead I marked the relevant symbols als
>   declspec(dllexport). I think we should do that regardless of the
>   buildsystem change. Restricting symbol visibility via gcc's
>   -fvisibility=hidden for extensions results in a substantially reduced
>   number of exported symbols, and even reduces object size (and I think
>   improves the code too).  I'll send an email about that separately.
>
 >
>
>
> There's a lot more stuff to talk about, but I'll stop with a small bit
> of instructions below:
>
>
> Demo / instructions:
> # Get code
> git remote add andres git@github.com:anarazel/postgres.git
> git fetch andres
> git checkout --track andres/meson
>
> # setup build directory
> meson setup build --buildtype debug
> cd build
>
> # build (uses automatically as many cores as available)
> ninja

I'm getting errors at this step. You can find my output at
https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
expected for now?

> # change configuration, build again
> meson configure -Dssl=openssl
> ninja
>
> # run all tests
> meson test
>
> # run just recovery tests
> meson test --suite setup --suite recovery
>
> # list tests
> meson test --list
>
>
> Greetings,
>
> Andres Freund

On 10/12/21 4:37 AM, Andres Freund wrote:
> # setup build directory
> meson setup build --buildtype debug

I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> writes:
> I think we're going to need some solution to this problem. We have too
> many people here with strong opinions about questions like this for me
> to feel good about the idea that we're going to collectively be OK
> with leaving these sorts of decisions up to some other project.

Agreed.  I'm willing to put up with the costs of moving to some
other build system, but not if it dictates choices we don't want to
make about the end products.

> From my point of view, the time it takes to run configure is annoying,
> but the build time is pretty fine. On my system, configure takes about
> 33 seconds, and a full rebuild with 'make -j8' takes 14.5 seconds (I
> am using ccache). Moreover, most of the time when I run make, I'm only
> doing a partial rebuild, so it's near-instantaneous.

Read about Autoconf's --cache-file option.  That and ccache are
absolutely essential tools IMO.

            regards, tom lane

On 10/12/21 11:28 AM, Andrew Dunstan wrote:
> On 10/12/21 4:37 AM, Andres Freund wrote:
>> # setup build directory
>> meson setup build --buildtype debug
> I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.
>
>

It hung because it expected the compiler to be 'ccache cc'. Hanging in
such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
but it then errored out looking for perl libs. I think msys2 is going to
be a bit difficult here :-(


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-12 15:30:57 +0200, Peter Eisentraut wrote:
> I played with $subject a few years ago and liked it.  I think, like you
> said, meson is the best way forward.  I support this project.

Cool.


> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in meson.

Yea, there's some of that. I think some degree of reduction in flexibility is
needed to realistically target multiple "backend" build-system like visual
studio project files etc. but I wish there were a bit less of that
nonetheless.


> For example, at the time, gcc on macOS was not supported.  Meson thought, if
> you are on macOS, you are surely using the Apple compiler, and it supports
> these options.

I'm pretty sure this one now can just be overridden with CC=gcc. It can on
linux and windows, but I don't have ready interactive access with a mac
(leaving cirrus asside, which now has a "start a terminal" option...).


> For example, the shared library build behavior has been carefully tuned in
> opinionated ways.  With the autotools chain, one can override anything with
> enough violence; so we have always felt free to do that. I haven't followed
> it in a while, so I don't know what the situation is now; but it is a
> concern, because we have always felt free to try new and unusual build tools
> (Sun compiler, Intel compiler, clang-when-it-was-new) early without waiting
> for anyone else.

It's possible to just take over building e.g. shared libraries ourselves with
custom targets. Although it'd be a bit annoying to do. The bigger problem is
that that e.g. wouldn't play that nicely with generating visual studio
projects, which require to generate link steps in a certain way. It'd build,
but the GUI might loose some of its options. Etc.

Greetings,

Andres Freund

Hi,

On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> It hung because it expected the compiler to be 'ccache cc'. Hanging in
> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> but it then errored out looking for perl libs. I think msys2 is going to
> be a bit difficult here :-(

Hm. Yea, the perl thing is my fault - you should be able to get past it with
-Dperl=disabled, and I'll take a look at fixing the perl detection. (*)

I can't reproduce the hanging though. I needed to install bison, flex and
ninja and disable perl as described above, but then it built just fine.

It does seems to crash somewhere in the main regression tests though, I think
I don't do the "set stack depth" dance correctly for msys.


If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?


(*) I've for now made most dependencies autodetected, unless you pass
--auto-features disabled to collectively disable all the auto-detected
features. Initially I had mirrored the autoconf behaviour, but I got sick of
forgetting to turn off readline or zlib on windows. And then it was useful to
test on multiple operating systems...

For working on windows meson's wraps are quite useful. I've not added that to
the git branch, but if you manually do
  mkdir subprojects
  meson wrap install lz4
  meson wrap install zlib
building with -Dzlib=enabled -Dlz4=enabled will fall back to building lz4,
zlib as-needed.

I was wondering about adding a binary wrap for e.g. bison, flex on windows, so
that the process of getting a build going isn't as arduous.

Greetings,

Andres Freund

Hi,

On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > # build (uses automatically as many cores as available)
> > ninja
> 
> I'm getting errors at this step. You can find my output at
> https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> expected for now?

Thanks, that's helpful. And no, that's not expected (*), it should be fixed.

What OS / distribution / version is this?

Can you build postgres "normally" with --with-gss? Seems like we're ending up
with a version of gssapi that we're not compatible with.

You should be able to get past this by disabling gss using meson configure
-Dgssapi=disabled.

Greetings,

Andres Freund

* except kinda, in the sense that I'd expect it to be buggy, given that I've
  run it only on a few machines and it's very, uh, bleeding edge

Hi,

On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> > It hung because it expected the compiler to be 'ccache cc'. Hanging in
> > such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> > but it then errored out looking for perl libs. I think msys2 is going to
> > be a bit difficult here :-(
> 
> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)

This is a weird one. I don't know much about msys, so it's probably related to
that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
shell commands that exists, but not according to msys's own python

$ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:',
os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False

$ ls -ld /usr/lib/perl5/core_perl/CORE
drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE

So it's not too surprising that that doesn't work out. It's easy enough to
work around, but still pretty weird.

I pushed a workaround for the config-time error, but it doesn't yet recognize
msys perl correctly. But at least it's not alone in that - configure doesn't
seem to either, so I'm probably doing something wrong :)


> I can't reproduce the hanging though. I needed to install bison, flex and
> ninja and disable perl as described above, but then it built just fine.
> 
> It does seems to crash somewhere in the main regression tests though, I think
> I don't do the "set stack depth" dance correctly for msys.

That was it - just hadn't ported setting -Wl,--stack=... for !msvc
windows. Pushed the fix for that out.


I guess I should figure out how to commandline install msys and add it to CI.

Greetings,

Andres Freund

On 10/12/21 12:59 PM, Andres Freund wrote:
>
>
> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?



Here's the entire thing


# cat 
C:/tools/msys64/home/Administrator/postgresql/build/meson-logs/meson-log.txt
Build started at 2021-10-12T18:08:34.387568
Main binary: C:/tools/msys64/mingw64/bin/python.exe
Build Options: -Dbuildtype=debug
Python system: Windows
The Meson build system
Version: 0.59.1
Source dir: C:/tools/msys64/home/Administrator/postgresql
Build dir: C:/tools/msys64/home/Administrator/postgresql/build
Build type: native build
Project name: postgresql
Project version: 15devel
Sanity testing C compiler: ccache cc
Is cross compiler: False.
Sanity check compiler command line: ccache cc sanitycheckc.c -o
sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----

meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
> On 10/12/21 12:59 PM, Andres Freund wrote:
> > If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?

> Here's the entire thing

> Sanity check compiler command line: ccache cc sanitycheckc.c -o
> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
> Sanity check compile stdout:
> 
> -----
> Sanity check compile stderr:
> 
> -----
> 
> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.

Huh, it's not a question of gcc vs cc, it's that meson automatically uses
ccache. And it looks like msys's ccache is broken at the moment (installed
yesterday):

$ ccache --version
ccache version 4.4.1
...

$ echo > test.c
$ ccache cc -c test.c
Segmentation fault (core dumped)
..

not sure how that leads to hanging, but it's not too surprising that things
don't work out after that...

Greetings,

Andres Freund

On 10/12/21 2:09 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>> but it then errored out looking for perl libs. I think msys2 is going to
>>> be a bit difficult here :-(
>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> This is a weird one. I don't know much about msys, so it's probably related to
> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> shell commands that exists, but not according to msys's own python
>
> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:',
os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> $ ls -ld /usr/lib/perl5/core_perl/CORE
> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE


Looks to me like a python issue:


# perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
exist: ), -e $p, qq{\n};'
does /usr/lib/perl5/core_perl/CORE exist: 1

# python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False

# cygpath -m /usr/lib/perl5/core_perl/CORE
C:/tools/msys64/usr/lib/perl5/core_perl/CORE

# python -c "import os; p =
'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
exist:', os.path.exists(p))"
does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True


Clearly python is not understanding msys virtualized paths.


>
>
> I guess I should figure out how to commandline install msys and add it to CI.
>


here's what I do:


    # msys2 outputs esc-[3J which clears the screen's scroll buffer. Nasty.
    # so we redirect the output
    # find the log in c:\Windows\System32 if needed
    choco install -y --no-progress --limit-output msys2 > msys2inst.log
    c:\tools\msys64\usr\bin\bash -l
    '/c/vfiles/windows-uploads/msys2-packages.sh'

Here's what's in msys-packages.sh:


    pacman -S --needed --noconfirm \
        base-devel \
        msys/git \
        msys/ccache \
        msys/vim  \
        msys/perl-Crypt-SSLeay \
        mingw-w64-clang-x86_64-toolchain \
        mingw-w64-x86_64-toolchain

    # could do: pacman -S --needed --noconfirm development
    # this is more economical. These should cover most of the things you
    might
    # want to configure with

    pacman -S --needed --noconfirm \
           msys/gettext-devel \
           msys/icu-devel \
           msys/libiconv-devel \
           msys/libreadline-devel \
           msys/libxml2-devel \
           msys/libxslt-devel \
           msys/openssl-devel \
           msys/zlib-devel


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 10/12/21 2:23 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
>> On 10/12/21 12:59 PM, Andres Freund wrote:
>>> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
>> Here's the entire thing
>> Sanity check compiler command line: ccache cc sanitycheckc.c -o
>> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
>> Sanity check compile stdout:
>>
>> -----
>> Sanity check compile stderr:
>>
>> -----
>>
>> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
> Huh, it's not a question of gcc vs cc, it's that meson automatically uses
> ccache. And it looks like msys's ccache is broken at the moment (installed
> yesterday):
>
> $ ccache --version
> ccache version 4.4.1
> ...
>
> $ echo > test.c
> $ ccache cc -c test.c
> Segmentation fault (core dumped)
> ..
>
> not sure how that leads to hanging, but it's not too surprising that things
> don't work out after that...
>

Yes, I've had to disable ccache on fairywren.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-12 09:15:41 -0700, Andres Freund wrote:
> > For example, at the time, gcc on macOS was not supported.  Meson thought, if
> > you are on macOS, you are surely using the Apple compiler, and it supports
> > these options.
>
> I'm pretty sure this one now can just be overridden with CC=gcc. It can on
> linux and windows, but I don't have ready interactive access with a mac
> (leaving cirrus asside, which now has a "start a terminal" option...).

It was a tad more complicated. But only because it took me a while to figure
out how to make gcc on macos actually work, independent of meson. Initially
gcc was always failing with errors about not finding the linker, and
installing binutils was a dead end.

Turns out just using a gcc at a specific path doesn't work, it ends up using
wrong internal binaries or something like that.

Once I got to that, the meson part was easy:

$ export PATH="/usr/local/opt/gcc/bin:$PATH"
$ CC=gcc-11 meson setup build-gcc
  ...
  C compiler for the host machine: gcc-11 (gcc 11.2.0 "gcc-11 (Homebrew GCC 11.2.0) 11.2.0")
  ...
$ cd build-gcc
$ ninja test
...

  181/181 postgresql:tap+subscription / subscription/t/100_bugs.pl                                 OK
17.83s  5 subtests passed
 


  Ok:                 180
  Expected Fail:      0
  Fail:               0
  Unexpected Pass:    0
  Skipped:            1
  Timeout:            0



One thing that is nice with meson's testrunner is that it can parse the output
of tap tests and recognizes the number of completed / failed subtests. I
wonder whether we could make pg_regress' output tap compliant without the
output quality suffering too much.

Greetings,

Andres Freund

On 10/12/21 2:37 PM, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>>> but it then errored out looking for perl libs. I think msys2 is going to
>>>> be a bit difficult here :-(
>>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
>> This is a weird one. I don't know much about msys, so it's probably related to
>> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
>> shell commands that exists, but not according to msys's own python
>>
>> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:',
os.path.exists(p))"
>> does /usr/lib/perl5/core_perl/CORE exist: False
>>
>> $ ls -ld /usr/lib/perl5/core_perl/CORE
>> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
>
> Looks to me like a python issue:
>
>
> # perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
> exist: ), -e $p, qq{\n};'
> does /usr/lib/perl5/core_perl/CORE exist: 1
>
> # python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
> print(f'does {p} exist:', os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> # cygpath -m /usr/lib/perl5/core_perl/CORE
> C:/tools/msys64/usr/lib/perl5/core_perl/CORE
>
> # python -c "import os; p =
> 'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
> exist:', os.path.exists(p))"
> does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True
>
>
> Clearly python is not understanding msys virtualized paths.


It's a matter of which python you use. The one that understands msys
paths is msys/python. The mingw64 packages are normally pure native
windows and so don't understand msys paths. I know it's confusing :-(


# /usr/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: True


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-12 14:37:04 -0400, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
> >> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> >> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> > This is a weird one. I don't know much about msys, so it's probably related to
> > that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> > shell commands that exists, but not according to msys's own python
> >
> > $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:',
os.path.exists(p))"
> > does /usr/lib/perl5/core_perl/CORE exist: False
> >
> > $ ls -ld /usr/lib/perl5/core_perl/CORE
> > drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE

> Looks to me like a python issue:

> Clearly python is not understanding msys virtualized paths.

Ah, it's a question of the *wrong* python being used :/. I somehow ended up
with both a mingw and an msys python, with the mingw python taking preference
over the msys one. The latter one does understand such paths.



> > I guess I should figure out how to commandline install msys and add it to CI.

> here's what I do:

Thanks!


Does that recipe get you to a build where ./configure --with-perl succeeds?

I see this here:

checking for Perl archlibexp... /usr/lib/perl5/core_perl
checking for Perl privlibexp... /usr/share/perl5/core_perl
checking for Perl useshrplib... true
checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64
-mtune=generic-O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
 
checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
checking for flags to link embedded Perl... no
configure: error: could not determine flags for linking embedded Perl.
This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
installed.

If I just include perl.h from a test file with gcc using the above flags it
fails to compile:
$ echo '#include <perl.h>' > test.c
$ gcc -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv
-fno-strict-aliasing-fstack-protector-strong test.c -c -I /c/dev/msys64/usr/lib/perl5/core_perl/CORE
 
In file included from test.c:1:
C:/dev/msys64/usr/lib/perl5/core_perl/CORE/perl.h:1003:13: fatal error: sys/wait.h: No such file or directory
 1003 | #   include <sys/wait.h>

and ldopts bleats

$ perl -MExtUtils::Embed -e ldopts
Warning (mostly harmless): No library found for -lpthread
Warning (mostly harmless): No library found for -ldl
   -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong
-L/usr/lib/perl5/core_perl/CORE-lperl -lcrypt
 

Greetings,

Andres Freund

On 10/12/21 3:29 PM, Andres Freund wrote:
>
> Does that recipe get you to a build where ./configure --with-perl succeeds?
>
> I see this here:
>
> checking for Perl archlibexp... /usr/lib/perl5/core_perl
> checking for Perl privlibexp... /usr/share/perl5/core_perl
> checking for Perl useshrplib... true
> checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64
-mtune=generic-O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
 
> checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
> checking for flags to link embedded Perl... no
> configure: error: could not determine flags for linking embedded Perl.
> This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
> installed.
>
> If I just include perl.h from a test file with gcc using the above flags it
> fails to compile:


You need to build against a native perl, like Strawberry or ActiveState.
(I have had mixed success with Strawberry) You do that by putting a path
to it at the start of the PATH. The wrinkle in this is that you need
prove to point to one that understands virtual paths. So you do
something like this:


PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...



cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> You need to build against a native perl, like Strawberry or ActiveState.
> (I have had mixed success with Strawberry)

Do you understand why that is needed?


> You do that by putting a path to it at the start of the PATH. The wrinkle in
> this is that you need prove to point to one that understands virtual
> paths. So you do something like this:
> 
> 
> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...

Oh my.

I'll try that later... I wonder if we could make this easier from our side?
This is a lot of magic to know.

Greetings,

Andres Freund

Hi,

On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:37 AM Andres Freund <andres@anarazel.de> wrote:
> The build code looks pretty approachable for someone with no prior
> exposure, and feels pretty nice when running it

That's part of what attracted me...


> (I couldn't get a build working but I'll leave that aside for now).

If you want to do that separately, I'll try to fix it.


> > As far as I can tell the only OS that postgres currently supports that
> > meson doesn't support is HPUX. It'd likely be fairly easy to add
> > gcc-on-hpux support, a chunk more to add support for the proprietary
> > ones.
> 
> That would also have to work for all the dependencies, which were displayed
> to me as:
> 
> ninja, gdbm, ca-certificates, openssl@1.1, readline, sqlite and python@3.9

meson does depend on ninja (to execute the build) and of course python. But
the rest should be optional dependencies. ninja builds without any
dependencies as long as you don't change its parser sources. python builds on
aix, hpux etc.

Not sure what way gdbm openssl@1.1 and sqlite are pulled in? I assume readline
is for python...


> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.

Yes, that shouldn't be a problem. You can run arbitrary code in targets
(there's plenty need for that already in what I have so far).

Greetings,

Andres Freund

út 12. 10. 2021 v 19:17 odesílatel Andres Freund <andres@anarazel.de> napsal:
>
> Hi,
>
> On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > > # build (uses automatically as many cores as available)
> > > ninja
> >
> > I'm getting errors at this step. You can find my output at
> > https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> > expected for now?
>
> Thanks, that's helpful. And no, that's not expected (*), it should be fixed.
>
> What OS / distribution / version is this?

Fedora 34 (64 bit)

> Can you build postgres "normally" with --with-gss? Seems like we're ending up
> with a version of gssapi that we're not compatible with.

Yes, I can.

> You should be able to get past this by disabling gss using meson configure
> -Dgssapi=disabled.

I tried to clean and start from scratch, but I'm getting different
error probably related to wrongly configured JIT (LLVM wasn't found
during meson setup). I'll debug on my side to provide more info.

Whole build error could be found at https://pastebin.com/hCFqcPvZ.
Setup log could be found at https://pastebin.com/wjbE1w56.

> Greetings,
>
> Andres Freund
>
> * except kinda, in the sense that I'd expect it to be buggy, given that I've
>   run it only on a few machines and it's very, uh, bleeding edge

Hi,

On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> I tried to clean and start from scratch, but I'm getting different
> error probably related to wrongly configured JIT (LLVM wasn't found
> during meson setup). I'll debug on my side to provide more info.

../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
   91 |         snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
      |                                                                         ^~~~~~~~

This *very* likely is related to building in a source tree that also contains
a "non-meson" build "in place". The problem is that the meson build picks up
the pg_config.h generated by ./configure in the "normal" build, rather than
the one meson generated itself.

You'd need to execute make distclean or such, or use a separate git checkout.

I forgot about this issue because I only ever build postgres from outside the
source-tree (by invoking configure from a separate directory), so there's
never build products in it. I think at least I need to make the build emit a
warning / error if there's a pg_config.h in the source tree...


This is the part of the jit code that's built regardless of llvm availability
- you'd get the same error in a few other places unrelated to jit.

Greetings,

Andres Freund

Hi,

On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> > You do that by putting a path to it at the start of the PATH. The wrinkle in
> > this is that you need prove to point to one that understands virtual
> > paths. So you do something like this:
> > 
> > 
> > PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
> 
> Oh my.
> 
> I'll try that later... I wonder if we could make this easier from our side?
> This is a lot of magic to know.

I managed to get this working. At first it failed because I don't have
pexports - it's not available inside msys as far as I could tell. And seems to
be unmaintained. But replacing pexports with gendef fixed that.

There's this comment in src/pl/plperl/GNUmakefile

# Perl on win32 ships with import libraries only for Microsoft Visual C++,
# which are not compatible with mingw gcc. Therefore we need to build a
# new import library to link with.

but I seem to be able to link fine without going through that song-and-dance?

Greetings,

Andres Freund

> On 12 Oct 2021, at 21:01, Andres Freund <andres@anarazel.de> wrote:

> One thing that is nice with meson's testrunner is that it can parse the output
> of tap tests and recognizes the number of completed / failed subtests. I
> wonder whether we could make pg_regress' output tap compliant without the
> output quality suffering too much.

I added a --tap option for TAP output to pg_regress together with Jinbao Chen
for giggles and killing some time a while back.  It's not entirely done and
sort of PoC, but most of it works.  Might not be of interest here, but in case
it is I've refreshed it slightly and rebased it.  There might be better ways to
do it, but the aim was to make the diff against the guts of pg_regress small
and instead extract output functions for the different formats.

It omits the test timings, but that could be added either as a diagnostic line
following each status or as a YAML block in TAP 13 (the attached is standard
TAP, not version 13 but the change would be trivial).

If it's helpful and there's any interest for this I'm happy to finish it up now.

One thing that came out of this, is that we don't really handle the ignored
tests in the way the code thinks it does for normal output, the attached treats
ignored tests as SKIP tests.

--
Daniel Gustafsson        https://vmware.com/

On 10/12/21 9:03 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
>> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
>>> You do that by putting a path to it at the start of the PATH. The wrinkle in
>>> this is that you need prove to point to one that understands virtual
>>> paths. So you do something like this:
>>>
>>>
>>> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
>> Oh my.
>>
>> I'll try that later... I wonder if we could make this easier from our side?
>> This is a lot of magic to know.
> I managed to get this working. At first it failed because I don't have
> pexports - it's not available inside msys as far as I could tell. And seems to
> be unmaintained. But replacing pexports with gendef fixed that.
>
> There's this comment in src/pl/plperl/GNUmakefile
>
> # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> # which are not compatible with mingw gcc. Therefore we need to build a
> # new import library to link with.
>
> but I seem to be able to link fine without going through that song-and-dance?
>


It looks like you're not building a native postgres, but rather one
targeted at msys. To build one that's native (i.e. runs without any
presence of msys) you need to do these things before building:

    MSYSTEM=MINGW64
    MSYSTEM_CHOST=x86_64-w64-mingw32
    PATH="/mingw64/bin:$PATH"

pexports will be in the resulting path, and the build will use the
native compiler.

You can use fairywren's config as a guide.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-13 11:51:03 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:59 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> > > (I couldn't get a build working but I'll leave that aside for now).
> >
> > If you want to do that separately, I'll try to fix it.
> 
> Okay, I pulled the latest commits and tried again:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:
> 
> [the last line is repeated a bunch of times, then...]

Oh. I actually saw that on CI at some point... That one is definitely
odd. Currently CI for OSX builds like

    - brew install make coreutils ccache icu4c lz4 tcl-tk openldap
    - brew install meson ninja python@3.9
..
        PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig:$PKG_CONFIG_PATH"
        PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"
        PKG_CONFIG_PATH="/usr/local/opt/openldap/lib/pkgconfig:$PKG_CONFIG_PATH"

        export PKG_CONFIG_PATH

        meson setup --buildtype debug -Dcassert=true -Dssl=openssl build

but I set that up knowing little about macos.


For the autoconf build CI currently does something similar via
        LIBS="/usr/local/lib:$LIBS"
        INCLUDES="/usr/local/include:$INCLUDES"
...
        LIBS="/usr/local/opt/openldap/lib:$LIBS"
        INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
        ...
          --with-includes="$INCLUDES" \
          --with-libs="$LIBS" \

are you doing something like that? Or does it work for you without? I vaguely
recall hitting a similar problem as you report when not passing
/usr/local/... to configure.


> i tried disabling ldap with the meson build but i'll spare the details of
> what went wrong there in case i did something wrong, so we can take things
> one step at a time.

you can change it for an existing builddir with
meson configure -dldap=disabled or when setting up a new builddir by passing
-dldap=disabled at that time.


> > > ninja, gdbm, ca-certificates, openssl@1.1, readline, sqlite and
> python@3.9
> >
> > meson does depend on ninja (to execute the build) and of course python.
> but
> > the rest should be optional dependencies. ninja builds without any
> > dependencies as long as you don't change its parser sources. python
> builds on
> > aix, hpux etc.
> >
> > not sure what way gdbm openssl@1.1 and sqlite are pulled in? i assume
> readline
> > is for python...
> 
> Hmm, weird.

They're homebrew python deps: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.9.rb#L28
which are optional things enabled explicitly:
https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.9.rb#L123

Greetings,

Andres Freund

Hi,

On 2021-10-13 08:55:38 -0400, Andrew Dunstan wrote:
> On 10/12/21 9:03 PM, Andres Freund wrote:
> > I managed to get this working. At first it failed because I don't have
> > pexports - it's not available inside msys as far as I could tell. And seems to
> > be unmaintained. But replacing pexports with gendef fixed that.
> >
> > There's this comment in src/pl/plperl/GNUmakefile
> >
> > # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> > # which are not compatible with mingw gcc. Therefore we need to build a
> > # new import library to link with.
> >
> > but I seem to be able to link fine without going through that song-and-dance?
> >
>
>
> It looks like you're not building a native postgres, but rather one
> targeted at msys. To build one that's native (i.e. runs without any
> presence of msys) you need to do these things before building:
>
>     MSYSTEM=MINGW64
>     MSYSTEM_CHOST=x86_64-w64-mingw32
>     PATH="/mingw64/bin:$PATH"

I had a config equivalent to this (slight difference in PATH, but the same gcc
being picked), and I just verified that it still works if I set up PATH like
that.  I get a working plperl out of it. Without msys on PATH or such.

where perl526.dll
C:\perl\strawberry-5.26.3.1-64bit\perl\bin\perl526.dll

dumpbin /imports 'C:/Users/anfreund/src/pg-meson/build-mingw/tmp_install/lib/plperl.dll'|grep dll

Dump of file C:\Users\anfreund\src\pg-meson\build-mingw\tmp_install\lib\plperl.dll
    KERNEL32.dll
    msvcrt.dll
    perl526.dll

dumpbin /imports .\build-mingw\tmp_install\bin\postgres.exe|grep dll
    ADVAPI32.dll
    KERNEL32.dll
    msvcrt.dll
    Secur32.dll
    WLDAP32.dll
    WS2_32.dll

do $$elog(NOTICE, "blob");$$ language plperl;
NOTICE:  blob
DO

To me this looks like it's a plperl built without the import file recreation,
without being msys dependent?


> pexports will be in the resulting path, and the build will use the
> native compiler.

I don't see pexports anywhere in the msys installation. I can see it available
on sourceforge, and I see a few others asking where to get it from in the
context of msys, and being pointed to manually downloading it.

Seems like we should consider using gendef instead of pexports, given it's
available in msys?

$ pacman -Fy
$ pacman -F gendef.exe
...
mingw64/mingw-w64-x86_64-tools-git 9.0.0.6316.acdc7adc9-1 (mingw-w64-x86_64-toolchain) [installed]
    mingw64/bin/gendef.exe
..
$ pacman -F pexports.exe
$ pacman -Fx pexports
<bunch of packages containing smtpexports.h>

Greetings,

Andres Freund

Hi,

On 2021-10-13 13:19:36 -0400, John Naylor wrote:
> On Wed, Oct 13, 2021 at 12:37 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > For the autoconf build CI currently does something similar via
> >         LIBS="/usr/local/lib:$LIBS"
> >         INCLUDES="/usr/local/include:$INCLUDES"
> > ...
> >         LIBS="/usr/local/opt/openldap/lib:$LIBS"
> >         INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
> >         ...
> >           --with-includes="$INCLUDES" \
> >           --with-libs="$LIBS" \
> >
> > are you doing something like that? Or does it work for you without? I
> vaguely
> > recall hitting a similar problem as you report when not passing
> > /usr/local/... to configure.
> 
> I didn't do anything like that for the autoconf build. I have in the past
> done things retail, like

I'll try to see how this works / what causes the breakage.


> Somehow our emails got lower-cased down here, but I tried it with capital D:

:)


> meson configure -Dldap=disabled
>
> inside the build dir and got this:
>
> ../meson.build:278:2: ERROR: Tried to assign the invalid value "None" of
> type NoneType to variable.
>
> Line 278 is
>
>   ldap_r = ldap = dependency('', required : false)

Oops, I broke that when trying to clean things up. I guess I write too much C
;). It needs to be two lines.

I pushed the fix for that.

Greetings,

Andres Freund

On 10/13/21 1:26 PM, Andres Freund wrote:
>
>> pexports will be in the resulting path, and the build will use the
>> native compiler.
> I don't see pexports anywhere in the msys installation. I can see it available
> on sourceforge, and I see a few others asking where to get it from in the
> context of msys, and being pointed to manually downloading it.



Weird. fairywren has it, which means that it must have been removed from
the packages at some stage, fairly recently as fairywren isn't that old.
I just confirmed the absence on a 100% fresh install.


It is in Strawberry's c/bin directory.


>
> Seems like we should consider using gendef instead of pexports, given it's
> available in msys?


Yeah. It's missing on my ancient msys animal (frogmouth), but it doesn't
build --with-perl.


jacana seems to have it.


If you prep a patch I'll test it.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
> If you prep a patch I'll test it.

Well, right now I'm wondering if the better fix is to just remove the whole
win32 block. I don't know how far back, but afaict it's not needed. Seems to
have been needed for narwhal at some point, according to 02b61dd08f99. But
narwhal is long dead.

Greetings,

Andres Freund

st 13. 10. 2021 v 1:54 odesílatel Andres Freund <andres@anarazel.de> napsal:
>
> Hi,
>
> On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> > I tried to clean and start from scratch, but I'm getting different
> > error probably related to wrongly configured JIT (LLVM wasn't found
> > during meson setup). I'll debug on my side to provide more info.
>
> ../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
>    91 |         snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
>       |                                                                         ^~~~~~~~
>
> This *very* likely is related to building in a source tree that also contains
> a "non-meson" build "in place". The problem is that the meson build picks up
> the pg_config.h generated by ./configure in the "normal" build, rather than
> the one meson generated itself.
>
> You'd need to execute make distclean or such, or use a separate git checkout.
>
> I forgot about this issue because I only ever build postgres from outside the
> source-tree (by invoking configure from a separate directory), so there's
> never build products in it. I think at least I need to make the build emit a
> warning / error if there's a pg_config.h in the source tree...

Hello, thanks for the hint. I can finally build using meson and run
regress tests.

The only problem I do have currently is auto-detection of perl. I'm
getting error related to missing "Opcode.pm". PERL is autodetected and
enabled (https://pastebin.com/xfRRrDcU).

I do get the same error when I enforce perl for current master build
(./configure --with-perl). Using ./configure with perl autodetection
skips plperl extension on my system.

Disabling perl manually for meson build (meson setup build
--reconfigure --buildtype debug -Dperl=disabled) works for me.

>
> This is the part of the jit code that's built regardless of llvm availability
> - you'd get the same error in a few other places unrelated to jit.
>
> Greetings,
>
> Andres Freund

On 10/13/21 5:46 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>> If you prep a patch I'll test it.
> Well, right now I'm wondering if the better fix is to just remove the whole
> win32 block. I don't know how far back, but afaict it's not needed. Seems to
> have been needed for narwhal at some point, according to 02b61dd08f99. But
> narwhal is long dead.
>
Ok, I'll test it out.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Thu, Oct 14, 2021 at 4:51 AM John Naylor
<john.naylor@enterprisedb.com> wrote:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10:
error:#include nested too deeply
 
> #include <ldap.h>
>          ^

I vaguely recall that PostgreSQL should build OK against Apple's copy
of OpenLDAP.  That recursive include loop is coming from a "framework"
header that contains just a couple of lines like #include <ldap.h> to
try to include the real header, which should also be in the include
path, somewhere like
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/ldap.h.
I think we'd need to figure out where that
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers
directive is coming from and get rid of it, so we can include the real
header directly.

Josef Šimánek <josef.simanek@gmail.com> writes:

> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).

Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
core module, and should be in /usr/lib64/perl5, judging by the paths in
the error message.

Which OS is this?  Some Linux distributions have separate packages for
the interpreter itself and the included modules, and the packages can be
named confusingly.  E.g. on older Redhat/Fedora versions you have to
install the 'perl-core' package to get all the modules, 'perl' is just
the interpreter and the bare minimum set of strictily necessary modules.

They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
minimal one.


- ilmari

čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
<ilmari@ilmari.org> napsal:
>
> Josef Šimánek <josef.simanek@gmail.com> writes:
>
> > The only problem I do have currently is auto-detection of perl. I'm
> > getting error related to missing "Opcode.pm". PERL is autodetected and
> > enabled (https://pastebin.com/xfRRrDcU).
>
> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> core module, and should be in /usr/lib64/perl5, judging by the paths in
> the error message.
>
> Which OS is this?  Some Linux distributions have separate packages for
> the interpreter itself and the included modules, and the packages can be
> named confusingly.  E.g. on older Redhat/Fedora versions you have to
> install the 'perl-core' package to get all the modules, 'perl' is just
> the interpreter and the bare minimum set of strictily necessary modules.
>
> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> minimal one.

I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
package. Anyway it behaves differently with autoconf tools and the
meson build system. Is perl disabled by default in the current build
system?

>
> - ilmari

On 2021-Oct-14, Josef Šimánek wrote:

> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package. Anyway it behaves differently with autoconf tools and the
> meson build system. Is perl disabled by default in the current build
> system?

Yes, you have to use --with-perl in order to get it.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"

Josef Šimánek <josef.simanek@gmail.com> writes:

> čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> <ilmari@ilmari.org> napsal:
>>
>> Josef Šimánek <josef.simanek@gmail.com> writes:
>>
>> > The only problem I do have currently is auto-detection of perl. I'm
>> > getting error related to missing "Opcode.pm". PERL is autodetected and
>> > enabled (https://pastebin.com/xfRRrDcU).
>>
>> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
>> core module, and should be in /usr/lib64/perl5, judging by the paths in
>> the error message.
>>
>> Which OS is this?  Some Linux distributions have separate packages for
>> the interpreter itself and the included modules, and the packages can be
>> named confusingly.  E.g. on older Redhat/Fedora versions you have to
>> install the 'perl-core' package to get all the modules, 'perl' is just
>> the interpreter and the bare minimum set of strictily necessary modules.
>>
>> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
>> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
>> minimal one.
>
> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package.`

Yes, it's a separate package, but the 'perl' package depends on all the
core module packages, so installing that should fix things.  You appear
to only have 'perl-interpreter' installed.

> Anyway it behaves differently with autoconf tools and the meson build
> system. Is perl disabled by default in the current build system?

configure doesn't auto-detect any optional features, they have to be
explicitly enabled using --with-foo switches.

- ilmari

Hi,

On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> On 2021-Oct-14, Josef Šimánek wrote:
>
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package. Anyway it behaves differently with autoconf tools and the
> > meson build system. Is perl disabled by default in the current build
> > system?

Hm, so it seems we should make the test separately verify that perl -M{Opcode,
ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
detection with a useful message?


> Yes, you have to use --with-perl in order to get it.

With the meson prototype I set most optional features to "auto", except for
LLVM, as that increases compile times noticeably.

For configure we didn't/don't want to do much auto-detection, because that
makes life harder for distributors. But meson has one switch controlling all
features set to 'auto' and not explicitly enabled/disabled:
  --auto-features {enabled,disabled,auto}                               Override value of all 'auto' features (default:
auto).
so the argument doesn't apply to the same degree there. We could default
auto-features to something else too.

There were two other reasons:

1) I got tired of needing to disable zlib, readline to be able to build on
   windows.
2) Exercising all the dependency detection / checking seems important at this
   stage

Greetings,

Andres Freund

Hi,

On 2021-10-13 23:58:12 +0200, Josef Šimánek wrote:
> st 13. 10. 2021 v 1:54 odesílatel Andres Freund <andres@anarazel.de> napsal:
> > This *very* likely is related to building in a source tree that also contains
> > a "non-meson" build "in place". The problem is that the meson build picks up
> > the pg_config.h generated by ./configure in the "normal" build, rather than
> > the one meson generated itself.
> >
> > You'd need to execute make distclean or such, or use a separate git checkout.
> >
> > I forgot about this issue because I only ever build postgres from outside the
> > source-tree (by invoking configure from a separate directory), so there's
> > never build products in it. I think at least I need to make the build emit a
> > warning / error if there's a pg_config.h in the source tree...
> 
> Hello, thanks for the hint. I can finally build using meson and run
> regress tests.

I yesterday pushed code that should detect this case (with an error). Should
now detect the situation both when you first run configure in tree, and then
meson, and the other way round (by the dirty hack of ./configure touch'ing
meson.build at the end for in-tree builds).


> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).
> 
> I do get the same error when I enforce perl for current master build
> (./configure --with-perl). Using ./configure with perl autodetection
> skips plperl extension on my system.
> 
> Disabling perl manually for meson build (meson setup build
> --reconfigure --buildtype debug -Dperl=disabled) works for me.

Yay, thanks for testing!

Greetings,

Andres Freund

čt 14. 10. 2021 v 15:32 odesílatel Dagfinn Ilmari Mannsåker
<ilmari@ilmari.org> napsal:
>
> Josef Šimánek <josef.simanek@gmail.com> writes:
>
> > čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> > <ilmari@ilmari.org> napsal:
> >>
> >> Josef Šimánek <josef.simanek@gmail.com> writes:
> >>
> >> > The only problem I do have currently is auto-detection of perl. I'm
> >> > getting error related to missing "Opcode.pm". PERL is autodetected and
> >> > enabled (https://pastebin.com/xfRRrDcU).
> >>
> >> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> >> core module, and should be in /usr/lib64/perl5, judging by the paths in
> >> the error message.
> >>
> >> Which OS is this?  Some Linux distributions have separate packages for
> >> the interpreter itself and the included modules, and the packages can be
> >> named confusingly.  E.g. on older Redhat/Fedora versions you have to
> >> install the 'perl-core' package to get all the modules, 'perl' is just
> >> the interpreter and the bare minimum set of strictily necessary modules.
> >>
> >> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> >> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> >> minimal one.
> >
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package.`
>
> Yes, it's a separate package, but the 'perl' package depends on all the
> core module packages, so installing that should fix things.  You appear
> to only have 'perl-interpreter' installed.

You're right. Installing "perl" or "perl-Opcode" manually fixes this
problem. Currently I only have "perl-interpreter" installed.

> > Anyway it behaves differently with autoconf tools and the meson build
> > system. Is perl disabled by default in the current build system?
>
> configure doesn't auto-detect any optional features, they have to be
> explicitly enabled using --with-foo switches.
>
> - ilmari

čt 14. 10. 2021 v 19:24 odesílatel Andres Freund <andres@anarazel.de> napsal:
>
> Hi,
>
> On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> > On 2021-Oct-14, Josef Šimánek wrote:
> >
> > > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > > package. Anyway it behaves differently with autoconf tools and the
> > > meson build system. Is perl disabled by default in the current build
> > > system?
>
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?

I can confirm "perl -MOpcode" fails. ExtUtils::Embed and
ExtUtils::ParseXS are present. Looking at the local system history of
perl-interpreter package, it seems to be installed by default on
Fedora 34. Friendly error message would be welcomed.

>
> > Yes, you have to use --with-perl in order to get it.
>
> With the meson prototype I set most optional features to "auto", except for
> LLVM, as that increases compile times noticeably.
>
> For configure we didn't/don't want to do much auto-detection, because that
> makes life harder for distributors. But meson has one switch controlling all
> features set to 'auto' and not explicitly enabled/disabled:
>   --auto-features {enabled,disabled,auto}                               Override value of all 'auto' features
(default:auto). 
> so the argument doesn't apply to the same degree there. We could default
> auto-features to something else too.
>
> There were two other reasons:
>
> 1) I got tired of needing to disable zlib, readline to be able to build on
>    windows.
> 2) Exercising all the dependency detection / checking seems important at this
>    stage

Clear, thanks for the info.

> Greetings,
>
> Andres Freund

Hi,

On October 14, 2021 12:14:16 PM PDT, John Naylor <john.naylor@enterprisedb.com> wrote:
>I wrote:
>
>> Ok great, it builds now! :-) Now something's off with dynamic loading.
>There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
>doesn't know to look for them there:
>>
>> $ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
>> dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
>>   Referenced from:
>/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
>>   Reason: image not found
>
>After poking a bit more, this only happens when trying to run the tests. If
>I specify a prefix, I can install, init, and start the server just fine, so
>that much works.

Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there.
Probablyneed to implement whatever Tom figured out to do about that for the current way of running tests. 

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hi,

On 14.10.2021 23:54, John Naylor wrote:
> On Thu, Oct 14, 2021 at 4:34 PM Andres Freund <andres@anarazel.de 
> <mailto:andres@anarazel.de>> wrote:
> 
>  > Is this a Mac with SIP enabled? The Mac CI presumably has that 
> disabled, which is why I didn't see this issue there. Probably need to 
> implement whatever Tom figured out to do about that for the current way 
> of running tests.
> 
> System Information says it's disabled. Running "csrutil status" 
> complains of an unsupported configuration, which doesn't sound good, so 
> I should probably go fix that independent of anything else. :-/


Maybe you could check that DYLD_LIBRARY_PATH is working for you?

% DYLD_FALLBACK_LIBRARY_PATH= 
DYLD_LIBRARY_PATH=./tmp_install/usr/local/lib 
./tmp_install/usr/local/bin/psql --version
psql (PostgreSQL) 15devel


Without DYLD_LIBRARY_PATH I get the error, as expected:

% DYLD_FALLBACK_LIBRARY_PATH= ./tmp_install/usr/local/bin/psql --version
dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
   Referenced from: 
/Users/shinderuk/src/postgres-meson/build/./tmp_install/usr/local/bin/psql
   Reason: image not found


I add "DYLD_FALLBACK_LIBRARY_PATH=" because otherwise dyld falls back to 
/usr/lib/libpq.5.dylib provided by Apple (I am testing on Catalina).

% DYLD_PRINT_LIBRARIES=1 ./tmp_install/usr/local/bin/psql --version 2>&1 
| grep libpq
dyld: loaded: <4EDF735E-2104-32AD-BE7B-B400ABFCF57C> /usr/lib/libpq.5.dylib


Regards,

-- 
Sergey Shinderuk        https://postgrespro.com/

John Naylor <john.naylor@enterprisedb.com> writes:
>> System Information says it's disabled. Running "csrutil status" complains
>> of an unsupported configuration, which doesn't sound good, so I should
>> probably go fix that independent of anything else. :-/

> Looking online, I wonder if the "unsupported" message might be overly
> cautious. In any case, I do remember turning something off to allow a
> debugger to run. Here are all the settings, in case it matters:

> Apple Internal: disabled
> Kext Signing: enabled
> Filesystem Protections: enabled
> Debugging Restrictions: disabled
> DTrace Restrictions: enabled
> NVRAM Protections: enabled
> BaseSystem Verification: enabled

I remember having seen that report too, after some previous software
upgrade that had started from a "SIP disabled" status.  I'm mostly
guessing here, but my guess is that

(a) csrutil only considers the all-enabled and all-disabled states
of these individual flags to be "supported" cases.

(b) some one or more of these flags came along in a macOS update,
and if you did the update starting from a "disabled" state, you
nonetheless ended up with the new flags enabled, leading to the
mixed state that csrutil complains about.

I've lost count of the number of times I've seen macOS updates
be sloppy about preserving non-default settings, so I don't find
theory (b) to be even slightly surprising.

Whether the mixed state is actually problematic in any way,
I dunno.  I don't recall having had any problems before noticing
that that was what I had.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there.
Probablyneed to implement whatever Tom figured out to do about that for the current way of running tests. 

AFAIR the only cases we've made work are

(1) disable SIP

(2) avoid the need for (1) by always doing "make install" before
"make check".

Peter E. did some hacking towards another solution awhile ago,
but IIRC it involved changing the built binaries, and I think
we concluded that the benefits didn't justify that.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?

Our existing policy is that we should check this at configure time,
not later.  Since plperl won't work at all without Opcode, it seems
appropriate to add a check there if you say --with-perl.  I wasn't
aware that Red Hat had unbundled that from the minimal perl
installation :-(.

OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
I doubt it's worth the configure cycles to check for those separately.

            regards, tom lane

On 10/13/21 7:11 PM, Andrew Dunstan wrote:
> On 10/13/21 5:46 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>>> If you prep a patch I'll test it.
>> Well, right now I'm wondering if the better fix is to just remove the whole
>> win32 block. I don't know how far back, but afaict it's not needed. Seems to
>> have been needed for narwhal at some point, according to 02b61dd08f99. But
>> narwhal is long dead.
>>
> Ok, I'll test it out.
>

confirmed that jacana doesn't need this code to build or test plperl
(all I did was change the test from win32 to win32x). There would still
be work to do to fix the contrib bool_plperl, jsonb_plperl and
hstore_plperl modules.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Peter E. did some hacking towards another solution awhile ago,
> but IIRC it involved changing the built binaries, and I think
> we concluded that the benefits didn't justify that.

Yeah, by now there are lots of useful blogs from various projects
figuring out that you can use the install_name_tool to adjust the
paths it uses to be absolute or relative to certain magic words, like
@executable_path/../lib/blah.dylib, which is tempting, but...
realistically, for serious hacking on a Mac, SIP is so annoying that
it isn't the only reason you'll want to turn it off: it stops
dtrace/dtruss/... from working, and somehow prevents debuggers from
working when you've ssh'd in from a remote machine with a proper
keyboard, and probably more things that I'm forgetting.

I wish I could find the Xnu source that shows exactly how and when the
environment is suppressed in this way to understand better, but it
doesn't jump out of Apple's github; maybe it's hiding in closed source
machinery...

Thomas Munro <thomas.munro@gmail.com> writes:
> I wish I could find the Xnu source that shows exactly how and when the
> environment is suppressed in this way to understand better, but it
> doesn't jump out of Apple's github; maybe it's hiding in closed source
> machinery...

I recall that we figured out awhile ago that the environment gets trimmed
when make (or whatever) executes some command via the shell; seemingly,
Apple has decided that /bin/sh is a security-critical program that mustn't
be run with a non-default DYLD_LIBRARY_PATH.  Dunno if that helps you
find where the damage is done exactly.

(The silliness of this policy, when you pair it with the fact that they
don't reset PATH at the same time, seems blindingly obvious to me.  But
apparently not to Apple.)

            regards, tom lane

Hi,

On 2021-10-14 18:00:49 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue
there.Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
 
> 
> AFAIR the only cases we've made work are
> 
> (1) disable SIP
> 
> (2) avoid the need for (1) by always doing "make install" before
> "make check".

Ah, I thought it was more than that. In that case, John, does meson's test
succeed after you did the "proper" install? Assuming it's in a path that's
allowed to provide shared libraries?

Greetings,

Andres Freund

I wrote:
> I recall that we figured out awhile ago that the environment gets trimmed
> when make (or whatever) executes some command via the shell; seemingly,
> Apple has decided that /bin/sh is a security-critical program that mustn't
> be run with a non-default DYLD_LIBRARY_PATH.  Dunno if that helps you
> find where the damage is done exactly.

BTW, here's the evidence for this theory:

[tgl@pro ~]$ cat checkenv.c
#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char **argv)
{
  char *pth = getenv("DYLD_LIBRARY_PATH");

  if (pth)
    printf("DYLD_LIBRARY_PATH = %s\n", pth);
  else
    printf("DYLD_LIBRARY_PATH is unset\n");

  return 0;
}
[tgl@pro ~]$ gcc checkenv.c
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ sh -c ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ bash -c ./a.out
DYLD_LIBRARY_PATH is unset

You have to check the environment using an "unprivileged" program.
If you try to examine the environment using, say, "env", you will get
very misleading results.  AFAICT, /usr/bin/env is *also* considered
security-critical, because I cannot get it to ever report that
DYLD_LIBRARY_PATH is set.

Hmm ... /usr/bin/perl seems to act the same way.  It can see
ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.

This may indicate that they've applied this policy on a blanket
basis to everything in /bin and /usr/bin (and other system
directories, maybe), rather than singling out the shell.

            regards, tom lane

Hi,

On 2021-10-15 11:23:00 +1300, Thomas Munro wrote:
> On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Peter E. did some hacking towards another solution awhile ago,
> > but IIRC it involved changing the built binaries, and I think
> > we concluded that the benefits didn't justify that.
> 
> Yeah, by now there are lots of useful blogs from various projects
> figuring out that you can use the install_name_tool to adjust the
> paths it uses to be absolute or relative to certain magic words, like
> @executable_path/../lib/blah.dylib, which is tempting, but...
> realistically, for serious hacking on a Mac, SIP is so annoying that
> it isn't the only reason you'll want to turn it off: it stops
> dtrace/dtruss/... from working, and somehow prevents debuggers from
> working when you've ssh'd in from a remote machine with a proper
> keyboard, and probably more things that I'm forgetting.

Meson has support for using install_name_tool to remove "build time" rpaths
and set "install time" rpaths during the installation process - which uses
install_name_tool on mac.

If, and perhaps that's too big an if, relative rpaths actually work despite
SIP, it might be worth setting a relative install_rpath, because afaict that
should then work both for a "real" installation and our temporary test one.

If absolute rpaths are required, it'd make the process a bit more expensive,
because we'd probably need to change a configure time option during the temporary
install. No actual rebuilds would be required, but still.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> If, and perhaps that's too big an if, relative rpaths actually work despite
> SIP, it might be worth setting a relative install_rpath, because afaict that
> should then work both for a "real" installation and our temporary test one.

From what we know so far, it seems like SIP wouldn't interfere with
that (if it works at all).  I think what SIP desires to prevent is
messing with a program's execution by setting DYLD_LIBRARY_PATH.
As long as the program executable itself is saying where to find
the library, I don't see why they should interfere with that.

(Again, it seems blindingly stupid to forbid this while not blocking
PATH or any of the other environment variables that have always affected
execution.  But what do I know.)

> If absolute rpaths are required, it'd make the process a bit more expensive,

It'd also put the kibosh on relocatable install trees, though I dunno how
much people really care about that.

            regards, tom lane

Hi,

On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> > ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> > detection with a useful message?
>
> Our existing policy is that we should check this at configure time,
> not later.

Yea, I was thinking of configure (and meson's equivalent) as well.


> Since plperl won't work at all without Opcode, it seems
> appropriate to add a check there if you say --with-perl.  I wasn't
> aware that Red Hat had unbundled that from the minimal perl
> installation :-(.
>
> OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
> I doubt it's worth the configure cycles to check for those separately.

On debian the perl binary, with a sparse set of modules is in
perl-base. ExtUtils::Embed and ExtUtils::ParseXS are in
perl-modules-x.yy. Whereas Opcode is in libperlx.yy. But libperlx.yy depends
on perl-modules-x.yy so I guess an Opcode.pm check would suffice.

Seems we can just check all of them at once with with something like

perl -MOpcode -MExtUtils::Embed -MExtUtils::ParseXSNotAvailable -e ''
Can't locate ExtUtils/ParseXSNotAvailable.pm in @INC (you may need to install the ExtUtils::ParseXS3 module) (@INC
contains:/home/andres/bin/perl5/lib/perl5/x86_64-linux-gnu-thread-multi /home/andres/bin/perl5/lib/perl5 /etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.32.1/usr/local/share/perl/5.32.1 /usr/lib/x86_64-linux-gnu/perl5/5.32
/usr/share/perl5/usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 /usr/share/perl/5.32
/usr/local/lib/site_perl).

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
>>> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
>>> detection with a useful message?

>> Our existing policy is that we should check this at configure time,
>> not later.

> Yea, I was thinking of configure (and meson's equivalent) as well.

Ah, sorry, I misunderstood what you meant by "test".

            regards, tom lane

Hi,

On 2021-10-14 19:27:17 -0400, John Naylor wrote:
> On Thu, Oct 14, 2021 at 6:55 PM Andres Freund <andres@anarazel.de> wrote:
> > Ah, I thought it was more than that. In that case, John, does meson's test
> > succeed after you did the "proper" install? Assuming it's in a path that's
> > allowed to provide shared libraries?
> 
> Oh, it can act like installcheck? [checks] Yep, "meson test" ran fine (*).
> It still ran the temp install first, but in any case it worked.

As far as I understand Tom, our normal make check only works on OSX if
previously you ran make install. Which will have installed libpq into the
"proper" install location. Because all our binaries will, by default, have an
rpath to the library directory embedded, that then allows binaries in the
temporary install to work. But using the wrong libpq - which most of the time
turns out to be harmless, because libpq doesn't change that rapidly.


> * (all passed but skipped subscription/t/012_collation.pl)

That test requires ICU, so that's fine. I guess we could prevent the test from
being executed in the first place, but I don't think we've done that for cases
where it's one specific test in a t/ directory, where others in the same
directory do not have such dependencies.

Greetings,

Andres Freund

On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> [tgl@pro ~]$ cat checkenv.c
> #include <stdio.h>
> #include <stdlib.h>
>
> int
> main(int argc, char **argv)
> {
>   char *pth = getenv("DYLD_LIBRARY_PATH");
>
>   if (pth)
>     printf("DYLD_LIBRARY_PATH = %s\n", pth);
>   else
>     printf("DYLD_LIBRARY_PATH is unset\n");
>
>   return 0;
> }
> [tgl@pro ~]$ gcc checkenv.c
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ sh -c ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ bash -c ./a.out
> DYLD_LIBRARY_PATH is unset
>
> You have to check the environment using an "unprivileged" program.
> If you try to examine the environment using, say, "env", you will get
> very misleading results.  AFAICT, /usr/bin/env is *also* considered
> security-critical, because I cannot get it to ever report that
> DYLD_LIBRARY_PATH is set.
>
> Hmm ... /usr/bin/perl seems to act the same way.  It can see
> ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.
>
> This may indicate that they've applied this policy on a blanket
> basis to everything in /bin and /usr/bin (and other system
> directories, maybe), rather than singling out the shell.

Looks like it.  If I've found the right code here, it looks like where
any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
for a setuid binary, they've trained theirs to whack DYLD_*, and also
for code-signed and __RESTRICT-marked executables.

https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681

I suppose you could point SHELL at an unsigned copy of sh (codesign
--remove-signature, or something from brew/ports/x) so that GNU make
should respect, but I don't know how many other exec("/bin/sh") calls
might be hiding around the place (I guess perl calls system()?) and
might require some kind of LD_PRELOAD hackery... not much fun.

Thomas Munro <thomas.munro@gmail.com> writes:
> On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> This may indicate that they've applied this policy on a blanket
>> basis to everything in /bin and /usr/bin (and other system
>> directories, maybe), rather than singling out the shell.

> Looks like it.  If I've found the right code here, it looks like where
> any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
> for a setuid binary, they've trained theirs to whack DYLD_*, and also
> for code-signed and __RESTRICT-marked executables.
> https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681

Ugh.  That explains it, all right.

> I suppose you could point SHELL at an unsigned copy of sh (codesign
> --remove-signature, or something from brew/ports/x) so that GNU make
> should respect, but I don't know how many other exec("/bin/sh") calls
> might be hiding around the place (I guess perl calls system()?) and
> might require some kind of LD_PRELOAD hackery... not much fun.

Yeah.  I thought about invoking everything via a small wrapper
that restores the correct setting of DYLD_LIBRARY_PATH.  We could
perhaps make that work for the invocations of test postmasters
and psqls from "make" and TAP scripts, but hacking up our code's
sundry uses of system(3) like that seems like it'd be very messy,
if feasible at all.

BTW, the POSIX spec explicitly discourages letting SHELL affect the
behavior of system(3), so I bet that wouldn't help.

            regards, tom lane

Hi,

On 2021-10-14 22:46:07 -0400, Tom Lane wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > I suppose you could point SHELL at an unsigned copy of sh (codesign
> > --remove-signature, or something from brew/ports/x) so that GNU make
> > should respect, but I don't know how many other exec("/bin/sh") calls
> > might be hiding around the place (I guess perl calls system()?) and
> > might require some kind of LD_PRELOAD hackery... not much fun.
> 
> Yeah.  I thought about invoking everything via a small wrapper
> that restores the correct setting of DYLD_LIBRARY_PATH.  We could
> perhaps make that work for the invocations of test postmasters
> and psqls from "make" and TAP scripts, but hacking up our code's
> sundry uses of system(3) like that seems like it'd be very messy,
> if feasible at all.

It does sound like using relative rpaths might be the thing we want - and like
they've been available since 10.5 or something.

Is there a reason we're using absolute rpaths on a bunch of platforms, rather
than relative ones, which'd allow relocation?

Greetings,

Andres Freund

Hi,

On 2021-10-14 19:23:58 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > If, and perhaps that's too big an if, relative rpaths actually work despite
> > SIP, it might be worth setting a relative install_rpath, because afaict that
> > should then work both for a "real" installation and our temporary test one.
> 
> From what we know so far, it seems like SIP wouldn't interfere with
> that (if it works at all).  I think what SIP desires to prevent is
> messing with a program's execution by setting DYLD_LIBRARY_PATH.
> As long as the program executable itself is saying where to find
> the library, I don't see why they should interfere with that.

Well, there's *some* danger with relative rpaths, because they might
accidentally be pointing somewhere non-existing and user-creatable. Not a huge
risk, but as you say:

> (Again, it seems blindingly stupid to forbid this while not blocking
> PATH or any of the other environment variables that have always affected
> execution.  But what do I know.)

these aren't necessarily carefuly weighed considerations :/

But it seems to work well from what I gather.


> > If absolute rpaths are required, it'd make the process a bit more expensive,
> 
> It'd also put the kibosh on relocatable install trees, though I dunno how
> much people really care about that.

We currently use absolute rpaths, or something equivalent.

The reason that running tests on macos works is that we set the "install_name"
of shared libraries to the intended installed location, using an absolute
path:
    LINK.shared        = $(COMPILER) -dynamiclib -install_name '$(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX)'
$(version_link)$(exported_symbols_list) -multiply_defined suppress
 
which on macos means that all libraries linking to that dylib reference it
under that absolute path.

On most other platforms we set an absolute rpath to the installation
directory, which has an equivalent effect:
rpathdir = $(libdir)


It seems to work quite well to change our own references to libpq in binaries
/ shared libs to be relative, but to leave the install_name of the libraries
intact. In combination with adding an rpath of @loader_path/../lib/ to
binaries and @loader_path/ to shlibs, the install will re relocatable.

It doesn't work as well to actually have a non-absolute install_name for
libraries (e.g. @rpath/libpq.dylib), because then external code linking to
libpq needs to add an rpath to the installation to make it work.

The advantage of this approach over Peter's is that it's not temp-install
specific - due to the relative paths, it makes installations relocatable
without relying [DY]LD_LIBRARY_PATH.

On other unixoid systems this whole mess is simpler, because we can just add
$ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
leave some absolute path in the libraries themself intact.

Greetings,

Andres Freund

Hi,

On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> It seems to work quite well to change our own references to libpq in binaries
> / shared libs to be relative, but to leave the install_name of the libraries
> intact. In combination with adding an rpath of @loader_path/../lib/ to
> binaries and @loader_path/ to shlibs, the install will re relocatable.
> 
> It doesn't work as well to actually have a non-absolute install_name for
> libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> libpq needs to add an rpath to the installation to make it work.
> 
> The advantage of this approach over Peter's is that it's not temp-install
> specific - due to the relative paths, it makes installations relocatable
> without relying [DY]LD_LIBRARY_PATH.
> 
> On other unixoid systems this whole mess is simpler, because we can just add
> $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> leave some absolute path in the libraries themself intact.

I implemented this for the meson build, and it seems to work nicely. The macos
part was harder than I hoped due to the install_name stuff, which meson
doesn't solve.

https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09

After that the build directory is relocatale.


I don't immediately see a way to do this reasonably for the autoconf
build. We'd need a list of our own shared libraries from somewhere, and then
replace the references after building the objects?

Greetings,

Andres Freund

Hi,

On 2021-10-15 15:36:16 -0700, Andres Freund wrote:
> On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> > It seems to work quite well to change our own references to libpq in binaries
> > / shared libs to be relative, but to leave the install_name of the libraries
> > intact. In combination with adding an rpath of @loader_path/../lib/ to
> > binaries and @loader_path/ to shlibs, the install will re relocatable.
> > 
> > It doesn't work as well to actually have a non-absolute install_name for
> > libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> > libpq needs to add an rpath to the installation to make it work.
> > 
> > The advantage of this approach over Peter's is that it's not temp-install
> > specific - due to the relative paths, it makes installations relocatable
> > without relying [DY]LD_LIBRARY_PATH.
> > 
> > On other unixoid systems this whole mess is simpler, because we can just add
> > $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> > leave some absolute path in the libraries themself intact.
> 
> I implemented this for the meson build, and it seems to work nicely. The macos
> part was harder than I hoped due to the install_name stuff, which meson
> doesn't solve.
> 
> https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09
> 
> After that the build directory is relocatale.

Well, now that I think about it, it's still only relocatable in the sense that
postgres itself will continue to work. Outside code linking to e.g. libpq will
get the wrong path after relocation the source tree, due to the absolute
install_name.

But that doesn't seem solvable, unless we make the installed install_name to
be '@rpath/libpq...dylib' and require code linking to libpq to pass
-Wl,-rpath,/path/to/libpq when linking to libpq.

Greetings,

Andres Freund

Hi Tom,

On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.

Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
supporting HP's compiler on pa-risc a while ago.

As I said it'd probably not be too hard to add meson support for hpux on hppa,
it's probably just a few branches. But that'd require access somewhere. The
gcc compile farm does not have a hppa member anymore...

I did notice that gcc will declare hppa-hpux obsolete in gcc 12 and will
remove at some point:
"The hppa[12]*-*-hpux10* and hppa[12]*-*-hpux11* configurations targeting 32-bit PA-RISC with HP-UX have been obsoleted
andwill be removed in a future release."
 
https://gcc.gnu.org/gcc-12/changes.html

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
>> As far as I can tell the only OS that postgres currently supports that
>> meson doesn't support is HPUX. It'd likely be fairly easy to add
>> gcc-on-hpux support, a chunk more to add support for the proprietary
>> ones.

> Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> supporting HP's compiler on pa-risc a while ago.

Right.  I am still testing with gcc on HP-PA.  I'd kind of like to
keep it running just as an edge case for our spinlock support, but
I'm not sure that I want to do any huge amount of work on meson
to keep that going.

I do have a functioning OpenBSD installation on that machine, so
one alternative if the porting costs look too high is to replace
gaur with an OpenBSD animal.  However, last I checked, OpenBSD
was about half the speed of HPUX on that hardware, so I'm not
real eager to go that way.  gaur's already about the slowest
animal in the farm :-(

> As I said it'd probably not be too hard to add meson support for hpux on hppa,
> it's probably just a few branches. But that'd require access somewhere. The
> gcc compile farm does not have a hppa member anymore...

If you've got an idea where to look, I could add that to my
to-do queue.

In any case, I don't think we need to consider HPUX as a blocker
for the meson approach.  The value-add from keeping gaur going
probably isn't terribly much.  I'm more concerned about the
effort involved in getting meson going on some other old animals,
such as prairiedog.

            regards, tom lane

Hi,

On 2021-10-19 17:57:31 -0400, John Naylor wrote:
> I know this is still in the evaluation stage, but I did notice some
> discrepencies in the Flex flags. With the attached patch, the read-only
> data segment seems to match up pretty well now.

Good catch. I think I just copied them around...

I wish we had a bit more consistency in the flags, so we could centralize
them. Seems there's no reason to not use -p -p and -b everywhere?


I also need to make meson use our flex wrapper for the relevant versions... I
can see the warning that'd be fixed by it on macos CI. Will do that and push
it out to my github repo together with your changes.

Thanks!

Andres

Hi,

On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> >> As far as I can tell the only OS that postgres currently supports that
> >> meson doesn't support is HPUX. It'd likely be fairly easy to add
> >> gcc-on-hpux support, a chunk more to add support for the proprietary
> >> ones.
>
> > Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> > supporting HP's compiler on pa-risc a while ago.
>
> Right.  I am still testing with gcc on HP-PA.  I'd kind of like to
> keep it running just as an edge case for our spinlock support, but
> I'm not sure that I want to do any huge amount of work on meson
> to keep that going.

Makes sense.  While that does test an odd special case for our spinlock
implementation, it's also the only supported platform with that edge case, and
it seems extremely unlikely that there ever will be a new platform with such
odd/limited atomic operations.


> I do have a functioning OpenBSD installation on that machine, so
> one alternative if the porting costs look too high is to replace
> gaur with an OpenBSD animal. However, last I checked, OpenBSD
> was about half the speed of HPUX on that hardware, so I'm not
> real eager to go that way.  gaur's already about the slowest
> animal in the farm :-(

Yea, that doesn't sound enticing. Seems like we either should keep it running
on hp-ux or just drop parisc support?


> > As I said it'd probably not be too hard to add meson support for hpux on hppa,
> > it's probably just a few branches. But that'd require access somewhere. The
> > gcc compile farm does not have a hppa member anymore...
>
> If you've got an idea where to look, I could add that to my to-do queue.

It might even just work. Looks like meson does have pa-risc detection. While
it doesn't have any specifically for hpux, it just falls back to python's
sys.platform in that case. python3 -c 'import sys;print(sys.platform)'

meson generates output for ninja to execute (basically a faster make that's
partially faster by being much less flexible. Intended to be output by more
user-friendly buildsystems ). Ninja can be built by a minimal python script,
or with cmake. The former doesn't seem to have hpux support, the latter does I
think.
https://github.com/ninja-build/ninja

So it could be interesting to see if ninja builds.


I've not taught the PG meson the necessary stuff for a 32 bit build. So
there's no point is trying whether meson works that much. I'll try to do that,
and let you know.


> I'm more concerned about the effort involved in getting meson going on some
> other old animals, such as prairiedog.

Yea, that's an *old* OS version. One version too old to have support for
@rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
According to wikipedia 10.5 is the last version to support ppc.

Looks like python still supports building back to 10.4.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I wish we had a bit more consistency in the flags, so we could centralize
> them. Seems there's no reason to not use -p -p and -b everywhere?

I don't think we care enough about performance of most of the scanners
to make them all backup-free, so -1 to that idea.

We could possibly replace the command line switches with %option
entries in the files themselves.  But I think the reason we haven't
done so for -b is that the Makefile still needs to know about it
so as to know what to do with the lex.backup output file.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
>> I'm more concerned about the effort involved in getting meson going on some
>> other old animals, such as prairiedog.

> Yea, that's an *old* OS version. One version too old to have support for
> @rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
> According to wikipedia 10.5 is the last version to support ppc.

My notes say

    Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
    supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
    Cube's ability.  Wikipedia does suggest it's possible to run Leopard, but...
    https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware

I'm not sure that I have install media for 10.5 anymore, either --- ISTR
some machine's CD drive failing and not letting me get the CD back out.
If I did have it, I don't think there'd be a way to update past 10.5.0
(surely Apple no longer has those updaters on-line?), so on the whole
I think that path is a nonstarter.

I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
a buildfarm animal on it.

As with the HPPA, a potential compromise is to spin up some newer
BSD-ish system on it.  I agree that OSX 10.4 is uninteresting as a
software platform, but I'd like to keep 32-bit PPC represented in
the farm.

            regards, tom lane

Hi,

On 2021-10-19 17:31:22 -0700, Andres Freund wrote:
> I also need to make meson use our flex wrapper for the relevant versions... I
> can see the warning that'd be fixed by it on macos CI. Will do that and push
> it out to my github repo together with your changes.

That turned out to be more work than I anticipated, so I pushed your changes
out separately.

There's this bit in plflex.pl that talks about adjusting yywrap() for msvc. I
didn't implement that and didn't see any compilation problems. Looks like that
originally hails from 2011, in 08a0c2dabc3b9d59d72d7a79ed867b8e37d275a7

Hm. Seems not worth carrying forward unless it actually causes trouble?

Greetings,

Andres Freund

Hi,

On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
> My notes say
> 
>     Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
>     supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
>     Cube's ability.  Wikipedia does suggest it's possible to run Leopard, but...
>     https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware
> 
> I'm not sure that I have install media for 10.5 anymore, either --- ISTR
> some machine's CD drive failing and not letting me get the CD back out.
> If I did have it, I don't think there'd be a way to update past 10.5.0
> (surely Apple no longer has those updaters on-line?), so on the whole
> I think that path is a nonstarter.

That does indeed sound like a nonstarter.


> I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
> noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
> a buildfarm animal on it.

Understandable.


> As with the HPPA, a potential compromise is to spin up some newer
> BSD-ish system on it.  I agree that OSX 10.4 is uninteresting as a
> software platform, but I'd like to keep 32-bit PPC represented in
> the farm.

I assume the reason 32-bit PPC is interesting is that it's commonly run big
endian?

I wonder when it'll be faster to run 32bit ppc via qemu than natively :)

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
>> As with the HPPA, a potential compromise is to spin up some newer
>> BSD-ish system on it.  I agree that OSX 10.4 is uninteresting as a
>> software platform, but I'd like to keep 32-bit PPC represented in
>> the farm.

> I assume the reason 32-bit PPC is interesting is that it's commonly run big
> endian?

Aside from bit width and endianness, I believe it's a somewhat smaller
instruction set than the newer CPUs.

> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)

I think qemu would have a ways to go for that.  More to the point,
I've found that its emulation is not as precise as one might wish...

            regards, tom lane

Hi,

On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)

Freebsd didn't seem to want to boot, but surprisingly a debian buster image
started at least the installer without problems... Will probably take a while
to see if it actually works.

I assume to make it acceptable from a build-speed perspective one would have
to use distcc with the compiler running outside.

Greetings,

Andres Freund

Hi,

On 2021-10-19 19:41:56 -0700, Andres Freund wrote:
> On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> > I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
>
> Freebsd didn't seem to want to boot, but surprisingly a debian buster image
> started at least the installer without problems... Will probably take a while
> to see if it actually works.

The build was quite slow (cold ccache cache, only 1 cpu):
real    106m33.418s
user    86m36.363s
sys    17m33.830s

But the actual test time wasn't *too* bad, compared to the 32bit ppc animals

real    12m14.944s
user    0m51.622s
sys    0m44.743s

Greetings,

Andres Freund

Hi,

On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.

Implementing reformat-dat-files was trivial:
https://github.com/anarazel/postgres/commit/29c1ce1ad4731290714978da5ce81e99ef051bec


However, update-unicode is a bit harder.  Partially not directly because of
meson, but because update-unicode as-is afaict doesn't support VPATH builds,
and meson enforces those.

make update-unicode
...
make -C src/common/unicode update-unicode
'/usr/bin/perl' generate-unicode_norm_table.pl
Can't open perl script "generate-unicode_norm_table.pl": No such file or directory

It's not too hard to fix. See attached for the minimal stuff that I
immediately found to be needed. There's likely more,
e.g. src/backend/utils/mb/Unicode - but I didn't immediately see where that's
invoked from.


The slightly bigger issue making update-unicode work with meson is that meson
doesn't provide support for invoking build targets in specific directories
(because it doesn't map nicely to e.g. msbuild). But scripts like
src/common/unicode/generate-unicode_norm_table.pl rely on CWD. It's not hard
to work around that, but IMO it's better for such scripts to not rely on CWD.


Greetings,

Andres Freund

Hi,

Attached is an updated version of the meson patchset.

Changes:

- support for remaining binaries in src/bin, contrib modules

- nearly all tests, including src/test/modules etc, are integrated.

- quite a few more, but not yet all, optional dependencies (most are
  exercised in the included CI)

- runs tests on SIP enabled macos without needing a prior installation /
  installation is relocatable

- support for building docs.
  I couldn't get dbtoepub work in a vpath style build, so I changed that
  to also use pandoc. No idea if anybody uses the epub rules?

- 32bit x86 [1], 64bit aarch64 builds

- cross-building windows from linux works

- error when building with meson against a source tree with an in-tree
  autoconf build (leads to problems with pg_config.h etc)

- update-unicode, reformat-dat-files, expand-dat-files


Bigger missing pieces:

- pgxs (that's a *hard* one)

- NLS

- test / add support for platforms besides freebsd, linux, macos, windows

- remaining hardcoded configure tests (e.g. ACCEPT_TYPE_ARG*)

- win32 resource files only handled for two binaries, needs to be made
  more compact

- ecpg

- fixing up flex output

- truckloads of polishing

- some tests (e.g. pg_upgrade, because of the upcoming tap conversion,
  other tests that are shell scripts). Some tests are now run
  unconditionally that previously were opt-in.

- what exactly gets installed where

- a "dist" target

- fix "ldap" build on macos


Greetings,

Andres Freund

[1] I had not defined SIZEOF_SIZE_T. Surprisingly that still results in
a successful 64bit build, but not a successful 32bit build.

On 01.11.21 00:24, Andres Freund wrote:
> - remaining hardcoded configure tests (e.g. ACCEPT_TYPE_ARG*)

I think we can get rid of that one.

That test originally catered to some strange edge cases where the third 
argument was size_t that was not the same size as int.  That is long 
gone, if it ever really existed.  All systems currently of interest use 
either socklen_t or int, and socklen_t is always int.  (A few build farm 
animals report size_t, but they are all 32-bit.)

I think we can change the code to use socklen_t and add a simple check 
to typedef socklen_t as int if not available.  See attached patch.

Hi,

On 2021-11-04 19:17:05 +0100, Peter Eisentraut wrote:
> On 01.11.21 00:24, Andres Freund wrote:
> > - remaining hardcoded configure tests (e.g. ACCEPT_TYPE_ARG*)
> 
> I think we can get rid of that one.

Oh, nice!

I was somewhat confused by "unsigned int PASCAL" as a type.


> That test originally catered to some strange edge cases where the third
> argument was size_t that was not the same size as int.  That is long gone,
> if it ever really existed.  All systems currently of interest use either
> socklen_t or int, and socklen_t is always int.  (A few build farm animals
> report size_t, but they are all 32-bit.)

> diff --git a/src/include/c.h b/src/include/c.h
> index c8ede08273..7c790f557e 100644
> --- a/src/include/c.h
> +++ b/src/include/c.h
> @@ -408,6 +408,10 @@ typedef unsigned char bool;
>   * ----------------------------------------------------------------
>   */
>  
> +#ifndef HAVE_SOCKLEN_T
> +typedef socklen_t int;
> +#endif

I'd put this in port.h instead of c.h, or is there a reason not to do so?


Probably worth putting this in fairly soon independent of whether anything
happens wrt meson?

Greetings,

Andres Freund

On 04.11.21 19:48, Andres Freund wrote:
> Probably worth putting this in fairly soon independent of whether anything
> happens wrt meson?

OK, done. Let's see what happens. ;-)

On 01.11.21 00:24, Andres Freund wrote:
> Hi,
> 
> Attached is an updated version of the meson patchset.

Nanoreview:  I think the patch

Subject: [PATCH v5 11/16] meson: prereq: Handle DLSUFFIX in msvc builds
  similar to other build envs.

is good to go.  It's not clear why it's needed in this context, but it 
seems good in general to make these things more consistent.

Hi,

On 2021-11-10 11:07:02 +0100, Peter Eisentraut wrote:
> On 01.11.21 00:24, Andres Freund wrote:
> > Hi,
> > 
> > Attached is an updated version of the meson patchset.
> 
> Nanoreview:  I think the patch

Thanks for looking!


> Subject: [PATCH v5 11/16] meson: prereq: Handle DLSUFFIX in msvc builds
>  similar to other build envs.
> 
> is good to go.  It's not clear why it's needed in this context, but it seems
> good in general to make these things more consistent.

The way it was set between msvc and other builds is currently inconsistent
between msvc and other builds, by virtue of win32_port.h defining for msvc:

/* Things that exist in MinGW headers, but need to be added to MSVC */
#ifdef _MSC_VER
...
/* Pulled from Makefile.port in MinGW */
#define DLSUFFIX ".dll"


it'd have needed unnecessarily contorted logic to continue setting DLSUFFIX
via commandline for !msvc, given that the the meson stuff is the same for msvc
and !msvc.

Greetings,

Andres Freund

Hi,

FWIW, I tried building postgres on a few other operating systems using
meson, after I got access to the gcc compile farm. Here's the results:


- openbsd: Compiled fine. Hit one issue running tests:

  openbsd has *completely* broken $ORIGIN support. It uses CWD as $ORIGIN
  rpaths, which obviously breaks for binaries invoked via PATH. So there
  goes the idea to only use $ORIGIN to run tests. Still seems worth to use
  on other platforms, particularly because it works with SIP on macos

  I understand not supporting $ORIGIN at all. But implementing it this way
  seems insane.


  I also ran into some problems with the semaphore limits. I had to switch to
  USE_NAMED_POSIX_SEMAPHORES to make the tests pass at all.


- netbsd: Compiled fine after some minor fix. There's a bit more to fix around
  many libraries not being in the normal library directory, but in
  /usr/pkg/lib, which is not in the library search path (i.e. we need to add
  an rpath for that in a few more places).


- AIX: Compiled and basic postgres runs fine after a few fixes (big endian
  test, converting exports.txt into the right format). Doesn't yet
  successfully run more than trivial tests, because I didn't implement the
  necessary generation of import files for postgres, but that's just a bit of
  work.

  This is hampered by the fact that the vanilla postgres crashes for me. I
  haven't quite figured out what's the problem. Might be a system issue -
  lots of other tools, e.g. perl, segfault frequently.


  One important thing to call out: Meson has support for the AIX linker, but
  *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
  try). I don't know if we'd require adding support for xlc to meson - xlc is
  pretty buggy and it doesn't seem particularly crucial to support such an old
  crufty compiler on a platform that's not used to a significant degree?


Greetings,

Andres

Andres Freund <andres@anarazel.de> writes:
>   One important thing to call out: Meson has support for the AIX linker, but
>   *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
>   try). I don't know if we'd require adding support for xlc to meson - xlc is
>   pretty buggy and it doesn't seem particularly crucial to support such an old
>   crufty compiler on a platform that's not used to a significant degree?

While I have no particular interest in AIX or xlc specifically, I do
worry about us becoming a builds-on-gcc-or-workalikes-only project.
I suppose MSVC provides a little bit of a cross-check, but I don't
really like giving up on other compilers.  Discounting gcc+clang+MSVC
leaves just a few buildfarm animals, and the xlc ones are a significant
part of that population.  (In fact, unless somebody renews fossa/husky's
icc license, the three xlc animals will be an outright majority of
them, because wrasse and anole are the only other active animals with
non-mainstream compilers.)

Having said that, I don't plan to be the one trying to get meson
to add xlc support ...

            regards, tom lane

Hi,

On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> >   One important thing to call out: Meson has support for the AIX linker, but
> >   *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
> >   try). I don't know if we'd require adding support for xlc to meson - xlc is
> >   pretty buggy and it doesn't seem particularly crucial to support such an old
> >   crufty compiler on a platform that's not used to a significant degree?
>
> While I have no particular interest in AIX or xlc specifically, I do
> worry about us becoming a builds-on-gcc-or-workalikes-only project.
> I suppose MSVC provides a little bit of a cross-check, but I don't
> really like giving up on other compilers.  Discounting gcc+clang+MSVC
> leaves just a few buildfarm animals, and the xlc ones are a significant
> part of that population.

Yea, that's a reasonable concern. I wonder if there's some non-mainstream
compiler that actually works on, um, more easily available platforms that we
could utilize.


> (In fact, unless somebody renews fossa/husky's
> icc license, the three xlc animals will be an outright majority of
> them, because wrasse and anole are the only other active animals with
> non-mainstream compilers.)

It should probably be doable to get somebody to run another icc animal. Icc is
supported by meson, fwiw.


> Having said that, I don't plan to be the one trying to get meson
> to add xlc support ...

It'd probably not be too hard. But given that it's quite hard to get access to
AIX + xlc, I'm not sure it's something I want to propose. There's no resources
to run halfway regular tests on that I found...


It's good to make sure we're not growing too reliant on some compiler(s), but
imo only really makes sense if the alternative compilers are meaningfully
available and maintained.

Greetings,

Andres Freund

On Mon, Nov 15, 2021 at 2:23 PM Andres Freund <andres@anarazel.de> wrote:
> It's good to make sure we're not growing too reliant on some compiler(s), but
> imo only really makes sense if the alternative compilers are meaningfully
> available and maintained.

That's a sensible position. I do worry that with this proposed move
we're going to be giving up some of the flexibility that we have right
now. I'm not sure exactly what that means in practice. But make is
just a way of running shell commands, and so you can run any shell
commands you want. The concept of some compiler not being supported
isn't really a thing that even makes sense in a world that is powered
by make. With a big enough hammer you can run any commands you like,
including any compilation commands you like. The whole thing is likely
to be a bit crufty which is a downside, and you might spend more time
fiddling with it than you really want. But nothing is really ever
blocked.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <andres@anarazel.de> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > Having said that, I don't plan to be the one trying to get meson
> > to add xlc support ...
>
> It'd probably not be too hard. But given that it's quite hard to get access to
> AIX + xlc, I'm not sure it's something I want to propose. There's no resources
> to run halfway regular tests on that I found...

FWIW there's a free-as-in-beer edition of xlc for Linux (various
distros, POWER only) so you could use qemu, though of course there
will be differences WRT AIX especially around linking, and I suppose a
big part of that work would be precisely understanding stuff like
linker details.

It looks like we have two xlc 12.1 compilers in the farm, but those
compilers are EOL'd[1].  The current release is 16.1, and we have one
of those.  The interesting thing about 16.1 is that you can invoke it
as xlclang to get the new clang frontend and, I think, possibly use
more clang/gcc-ish compiler switches[2].

[1] https://www.ibm.com/support/pages/lifecycle/search?q=xl%20c%2Fc%2B%2B
[2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end

Thomas Munro <thomas.munro@gmail.com> writes:
> ... The interesting thing about 16.1 is that you can invoke it
> as xlclang to get the new clang frontend and, I think, possibly use
> more clang/gcc-ish compiler switches[2].
> [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end

Ho, that's an earful ;-).  Though I wonder whether that frontend
hides the AIX-specific linking issues you mentioned.  (Also, although
I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
So whether we have useful access to it right now is unclear.)

This plays into something that was nagging at me while I wrote my
upthread screed about not giving up on non-gcc/clang compilers:
are those compilers outcompeting all the proprietary ones, to the
extent that the latter will be dead soon anyway?  I think Microsoft
is rich enough and stubborn enough to keep on developing MSVC no
matter what, but other compiler vendors may see the handwriting
on the wall.  Writing C compilers can't be a growth industry these
days.

            regards, tom lane

Hi,

On 2021-11-15 17:34:33 -0500, Tom Lane wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > ... The interesting thing about 16.1 is that you can invoke it
> > as xlclang to get the new clang frontend and, I think, possibly use
> > more clang/gcc-ish compiler switches[2].
> > [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
>
> Ho, that's an earful ;-).  Though I wonder whether that frontend
> hides the AIX-specific linking issues you mentioned.  (Also, although
> I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
> So whether we have useful access to it right now is unclear.)

It's actually available there, but in /opt/IBM/xlC/16.1.0/bin/xlclang++ (note
the upper case C).

It doesn't really hide the linking issues afaict. I think they're basically an
ABI rather than a linker invocation issue. It's not that hard to address them
though, it's basically making mkldexport.sh a tiny bit more general and
integrating it into src/backend/postgres' build.

We don't have to generate export files for shared libraries anymore though,
afaict, because there's 'expall', which suffices for our purposes. dlopen()
doesn't require an import file.


> This plays into something that was nagging at me while I wrote my
> upthread screed about not giving up on non-gcc/clang compilers:
> are those compilers outcompeting all the proprietary ones, to the
> extent that the latter will be dead soon anyway?

I think that's a pretty clear trend. The ones that aren't dying seem to be
incrementally onto more and more rebasing onto llvm tooling.

It doesn't help that most of those compilers are primarily for OSs that, uh,
aren't exactly growing. Which limits their potential usability significantly.

Greetings,

Andres Freund

On Tue, Nov 16, 2021 at 11:08 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> FWIW there's a free-as-in-beer edition of xlc for Linux (various
> distros, POWER only) so you could use qemu,

(It's also known to be possible to run AIX 7.2 on qemu, but the
install media is not made available to developers for testing/CI
without a hardware serial number.  Boo.)

On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <andres@anarazel.de> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > (In fact, unless somebody renews fossa/husky's
> > icc license, the three xlc animals will be an outright majority of
> > them, because wrasse and anole are the only other active animals with
> > non-mainstream compilers.)
>
> It should probably be doable to get somebody to run another icc animal. Icc is
> supported by meson, fwiw.

FWIW, in case someone is interested in bringing ICC back to the farm,
some light googling tells me that newer editions of "classic" ICC (as
opposed to "data parallel" ICC, parts of some kind of rebrand) no
longer require regular licence bureaucracy, and can be installed in
modern easier to maintain ways.  For example, I see that some people
add Intel's APT repository and apt-get install the compiler inside CI
jobs, on Ubuntu.

On 10/13/21 16:06, Andrew Dunstan wrote:
> On 10/13/21 1:26 PM, Andres Freund wrote:
>>> pexports will be in the resulting path, and the build will use the
>>> native compiler.
>> I don't see pexports anywhere in the msys installation. I can see it available
>> on sourceforge, and I see a few others asking where to get it from in the
>> context of msys, and being pointed to manually downloading it.
>
>
> Weird. fairywren has it, which means that it must have been removed from
> the packages at some stage, fairly recently as fairywren isn't that old.
> I just confirmed the absence on a 100% fresh install.
>
>
> It is in Strawberry's c/bin directory.
>
>
>> Seems like we should consider using gendef instead of pexports, given it's
>> available in msys?
>
> Yeah. It's missing on my ancient msys animal (frogmouth), but it doesn't
> build --with-perl.
>
>
> jacana seems to have it.
>
>
> If you prep a patch I'll test it.
>
>

Here's a patch. I've tested the perl piece on master and it works fine.
It applies cleanly down to 9.4, which is before we got transform modules
(9.5) which fail if we just omit doing this platform-specific piece.


Before that only plpython uses pexports, and we're not committed to
supporting plpython at all on old branches.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2022-02-06 12:06:41 -0500, Andrew Dunstan wrote:
> Here's a patch. I've tested the perl piece on master and it works fine.
> It applies cleanly down to 9.4, which is before we got transform modules
> (9.5) which fail if we just omit doing this platform-specific piece.

Given https://postgr.es/m/34e972bc-6e75-0754-9e6d-cde2518773a1%40dunslane.net
wouldn't it make sense to simply remove the pexports/gendef logic instead of
moving to gendef?

Greetings,

Andres Freund

On 2/6/22 13:39, Andres Freund wrote:
> Hi,
>
> On 2022-02-06 12:06:41 -0500, Andrew Dunstan wrote:
>> Here's a patch. I've tested the perl piece on master and it works fine.
>> It applies cleanly down to 9.4, which is before we got transform modules
>> (9.5) which fail if we just omit doing this platform-specific piece.
> Given https://postgr.es/m/34e972bc-6e75-0754-9e6d-cde2518773a1%40dunslane.net
> wouldn't it make sense to simply remove the pexports/gendef logic instead of
> moving to gendef?
>

I haven't found a way to fix the transform builds if we do that. So
let's leave that as a separate exercise unless you have a solution for
that - this patch is really trivial.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

I've been wondering whether we should try to have the generated pg_config.h
look as similar as possible to autoconf/autoheader's, or not. And whether the
way autoconf/autoheader define symbols makes sense when not using either
anymore.

To be honest, I do not really understand the logic behind when autoconf ends
up with #defines that define a macro to 0/1 and when a macro ends defined/or
not and when we end up with a macro defined to 1 or not defined at all.

So far I've tried to mirror the logic, but not the description / comment
formatting of the individual macros.

The "defined to 1 or not defined at all" behaviour is a mildly awkward to
achieve with meson, because it doesn't match the behaviour for booleans
options meson has (there are two builtin behaviours, one to define/undefine a
macro, the other to set the macro to 0/1. But there's none that defines a
macro to 1 or undefines it).

Probably best to initially have the macros defined as similar as reasonably
possible, but subsequently clean things up a bit.


A second aspect that I'm wondering about is whether we should try to split
pg_config.h output a bit:

With meson it's easy to change options like whether to build with some
dependency in an existing build tree and then still get a reliable build
result (ninja rebuilds if the commandline changed from the last invocation).

But right now doing so often ends up with way bigger rebuilds than necessary,
because for a lot of options we add #define USE_LDAP 1 etc to pg_config.h,
which of course requires rebuilding a lot of files. Even though most of these
symbols are only checked in a handful of files, often only .c files.

It seems like it might make sense to separate out defines that depend on the
compiler / "standard libraries" (e.g. {SIZEOF,ALIGNOF}_*,
HAVE_DECL_{STRNLEN,...}, HAVE_*_H) from feature defines (like
USE_{LDAP,ICU,...}). The header containing the latter could then included in
the places needing it (or we could have one header for each of the places
using it).

Perhaps we should also separate out configure-time settings like BLCKSZ,
DEF_PGPORT, etc. Realistically most of them are going to require a "full tree"
recompile anway, but it seems like it might make things easier to understand.

I think a split into pg_config_{platform,features,settings}.h could make sense.

Similar to above, it's probably best to do this separately after merging meson
support. But knowing what the split should eventually look like would be
helpful before, to ensure it's easy to do.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I've been wondering whether we should try to have the generated pg_config.h
> look as similar as possible to autoconf/autoheader's, or not. And whether the
> way autoconf/autoheader define symbols makes sense when not using either
> anymore.

> To be honest, I do not really understand the logic behind when autoconf ends
> up with #defines that define a macro to 0/1 and when a macro ends defined/or
> not and when we end up with a macro defined to 1 or not defined at all.

Agreed, that always seemed entirely random to me too.  I'd be content
to end up with "defined or not defined" as the standard.  I think
we have way more #ifdef tests than #if tests, so changing the latter
seems more sensible than changing the former.

> A second aspect that I'm wondering about is whether we should try to split
> pg_config.h output a bit:

TBH I can't get excited about that.  I do not think that rebuilding
with different options is a critical path.  ccache already does most
of the heavy lifting when you do that sort of thing, anyway.

            regards, tom lane

Hi,

I was trying to fix a few perl embedding oddities in the meson
patchset.

Whenever I have looked at the existing code, I've been a bit confused about
the following

code/comment in perl.m4:

# PGAC_CHECK_PERL_EMBED_LDFLAGS
# -----------------------------
# We are after Embed's ldopts, but without the subset mentioned in
# Config's ccdlflags; [...]

    pgac_tmp1=`$PERL -MExtUtils::Embed -e ldopts`
    pgac_tmp2=`$PERL -MConfig -e 'print $Config{ccdlflags}'`
    perl_embed_ldflags=`echo X"$pgac_tmp1" | sed -e "s/^X//" -e "s%$pgac_tmp2%%" -e ["s/ -arch [-a-zA-Z0-9_]*//g"]`

What is the reason behind subtracting ccdlflags?


The comment originates in:

commit d69a419e682c2d39c2355105a7e5e2b90357c8f0
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   2009-09-08 18:15:55 +0000

    Remove any -arch switches given in ExtUtils::Embed's ldopts from our
    perl_embed_ldflags setting.  On OS X it seems that ExtUtils::Embed is
    trying to force a universal binary to be built, but you need to specify
    that a lot further upstream if you want Postgres built that way; the only
    result of including -arch in perl_embed_ldflags is some warnings at the
    plperl.so link step.  Per my complaint and Jan Otto's suggestion.

but the subtraction goes all the way back to

commit 7662419f1bc1a994193c319c9304dfc47e121c98
Author: Peter Eisentraut <peter_e@gmx.net>
Date:   2002-05-28 16:57:53 +0000

    Change PL/Perl and Pg interface build to use configured compiler and
    Makefile.shlib system, not MakeMaker.


Greetings,

Andres Freund

Hi,

On 2022-02-07 16:30:53 -0500, Tom Lane wrote:
> > A second aspect that I'm wondering about is whether we should try to split
> > pg_config.h output a bit:
> 
> TBH I can't get excited about that.  I do not think that rebuilding
> with different options is a critical path.  ccache already does most
> of the heavy lifting when you do that sort of thing, anyway.

I've found it to be pretty painful when building with msvc, which doesn't have
ccache (yet at least), and where the process startup overhead is bigger.

Even on some other platforms it's useful - it takes a while on net/openbsd to
recompile postgres, even if everything is in ccache. If I test on some
platform I'll often install the most basic set, get the tests to run, and then
incrementally figure out what other packages need to be installed etc.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> What is the reason behind subtracting ccdlflags?

It looks like the coding actually originated here:

commit f5d0c6cad5bb2706e0e63f3f8f32e431ea428100
Author: Bruce Momjian <bruce@momjian.us>
Date:   Wed Jun 20 00:26:06 2001 +0000

    Apparently, on some systems, ExtUtils::Embed and MakeMaker are slightly
    broken, and its impossible to make a shared library when compiling with
    both CCDLFLAGS and LDDLFAGS, you have to pick one or the other.

    Alex Pilosov

and Peter just copied the logic in 7662419f1.  Considering that
the point of 7662419f1 was to get rid of MakeMaker, maybe we no
longer needed that at that point.

On my RHEL box, the output of ldopts is sufficiently redundant
that the subtraction doesn't actually accomplish much:

$ perl -MExtUtils::Embed -e ldopts
-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld-fstack-protector-strong -L/usr/local/lib  -L/usr/lib64/perl5/CORE -lperl
-lpthread-lresolv -ldl -lm -lcrypt -lutil -lc 

$ perl -MConfig -e 'print $Config{ccdlflags}'
-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld

which leads to

$ grep perl_embed_ldflags src/Makefile.global
perl_embed_ldflags      =  -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
-fstack-protector-strong-L/usr/local/lib  -L/usr/lib64/perl5/CORE -lperl -lpthread -lresolv -ldl -lm -lcrypt -lutil -lc 

so the only thing we actually got rid of was -Wl,--enable-new-dtags,
which I think we'll put back anyway.

Things might be different elsewhere of course, but I'm tempted
to take out the ccdlflags subtraction and see what the buildfarm
says.

            regards, tom lane

I wrote:
> Andres Freund <andres@anarazel.de> writes:
>> What is the reason behind subtracting ccdlflags?

> It looks like the coding actually originated here:
> commit f5d0c6cad5bb2706e0e63f3f8f32e431ea428100

Ah, here's the thread leading up to that:

https://www.postgresql.org/message-id/flat/200106191206.f5JC6R108371%40candle.pha.pa.us

The use of ldopts rather than hand-hacked link options seems to date to
0ed7864d6, only a couple days before that.  I don't think we had a
buildfarm then, but I'd bet against the problem being especially
widespread even then, or more people would've complained.


BTW, the business with zapping arch options seems to not be necessary
anymore either on recent macOS:

$ perl -MExtUtils::Embed -e ldopts
   -fstack-protector-strong  -L/System/Library/Perl/5.30/darwin-thread-multi-2level/CORE -lperl
$ perl -MConfig -e 'print $Config{ccdlflags}'
 $

(same results on either Intel or ARM Mac).  However, it looks like it
is still necessary to keep locust happy, and I have no idea just when
Apple stopped using arch switches here, so we'd better keep that.

            regards, tom lane

Hi,

On 2022-02-07 20:42:09 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > What is the reason behind subtracting ccdlflags?
>
> It looks like the coding actually originated here:
>
> commit f5d0c6cad5bb2706e0e63f3f8f32e431ea428100
> Author: Bruce Momjian <bruce@momjian.us>
> Date:   Wed Jun 20 00:26:06 2001 +0000
>
>     Apparently, on some systems, ExtUtils::Embed and MakeMaker are slightly
>     broken, and its impossible to make a shared library when compiling with
>     both CCDLFLAGS and LDDLFAGS, you have to pick one or the other.
>
>     Alex Pilosov
>
> and Peter just copied the logic in 7662419f1.  Considering that
> the point of 7662419f1 was to get rid of MakeMaker, maybe we no
> longer needed that at that point.

Yea. And maybe what was broken in 2001 isn't broken anymore either ;)


Looking at a number of OSs:

debian sid:
embed:  -Wl,-E  -fstack-protector-strong -L/usr/local/lib  -L/usr/lib/x86_64-linux-gnu/perl/5.34/CORE -lperl -ldl -lm
-lpthread-lc -lcrypt 
ldopts: -Wl,-E

fedora:
embed:  -Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -fstack-protector-strong
-L/usr/local/lib -L/usr/lib64/perl5/CORE -lperl -lpthread -lresolv -ldl -lm -lcrypt -lutil -lc 
ldopts: -Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1

suse tumbleweed:
embed:  -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34.0/x86_64-linux-thread-multi/CORE  -L/usr/local/lib64
-fstack-protector-strong -L/usr/lib/perl5/5.34.0/x86_64-linux-thread-multi/CORE -lperl -lm -ldl -lcrypt -lpthread 
ldopts: -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34.0/x86_64-linux-thread-multi/CORE

freebsd:
embed:  -Wl,-R/usr/local/lib/perl5/5.30/mach/CORE -pthread -Wl,-E  -fstack-protector-strong -L/usr/local/lib
-L/usr/local/lib/perl5/5.30/mach/CORE-lperl -lpthread -lm -lcrypt -lutil 
ldopts: -Wl,-R/usr/local/lib/perl5/5.30/mach/CORE

netbsd:
embed:  -Wl,-E  -Wl,-R/usr/pkg/lib/perl5/5.34.0/x86_64-netbsd-thread-multi/CORE  -pthread -L/usr/lib -Wl,-R/usr/lib
-Wl,-R/usr/pkg/lib-L/usr/pkg/lib  -L/usr/pkg/lib/perl5/5.34.0/x86_64-netbsd-thread-multi/CORE -lperl -lm -lcrypt
-lpthread
ldopts: -Wl,-E  -Wl,-R/usr/pkg/lib/perl5/5.34.0/x86_64-netbsd-thread-multi/CORE

openbsd:
embed:  -Wl,-R/usr/libdata/perl5/amd64-openbsd/CORE -Wl,-E  -fstack-protector-strong -L/usr/local/lib
-L/usr/libdata/perl5/amd64-openbsd/CORE-lperl -lm -lc 
ldopts: -Wl,-R/usr/libdata/perl5/amd64-openbsd/CORE

aix:
embed:  -bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp
-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp-brtl -bdynamic -b64
-L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE-lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads
-lc
ldopts: -bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp
-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp

mac m1 monterey:
embed:  -fstack-protector-strong  -L/System/Library/Perl/5.30/darwin-thread-multi-2level/CORE -lperl
ldopts:

windows msys install ucrt perl:
embed:  -s -L"C:\dev\msys64\ucrt64\lib\perl5\core_perl\CORE" -L"C:\dev\msys64\ucrt64\lib"
"C:\dev\msys64\ucrt64\lib\perl5\core_perl\CORE\libperl532.a"
ldopts:

windows strawberrry perl:
embed:  -s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"  "C:\STRAWB~1\perl\lib\CORE\libperl530.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libmoldname.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libkernel32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libuser32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libgdi32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libwinspool.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libcomdlg32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libadvapi32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libshell32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libole32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\liboleaut32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libnetapi32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libuuid.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libws2_32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libmpr.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libwinmm.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libversion.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libodbc32.a""C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libodbccp32.a"
"C:\STRAWB~1\c\x86_64-w64-mingw32\lib\libcomctl32.a"
ldopts:


So on windows, macos it makes no difference because ldopts is empty.

On various linuxes, except red-hat and debian ones, as well as on the BSDs, it
removes rpath. Which we then add back in various places (pl and transform
modules). On debian the added rpath never will contain the library.

AIX is the one exception. Specifying -bE... doesn't seem right for building
plperl etc. So possibly the subtraction accidentally does work for us there...


> Things might be different elsewhere of course, but I'm tempted
> to take out the ccdlflags subtraction and see what the buildfarm
> says.

Except for the AIX thing I agree :(

Greetings,

Andres Freund

On 07.02.22 20:24, Andres Freund wrote:
> To be honest, I do not really understand the logic behind when autoconf ends
> up with #defines that define a macro to 0/1 and when a macro ends defined/or
> not and when we end up with a macro defined to 1 or not defined at all.

The default is to define to 1 or not at all.  The reason for this is 
presumably that originally, autoconf (or its predecessor practices) just 
populated the command line with a few -DHAVE_THIS options.  Creating a 
header file came later.  And -DFOO is equivalent to #define FOO 1. 
Also, this behavior allows code to use both the #ifdef HAVE_THIS and the 
#if HAVE_THIS style.

The cases that deviate from this have a special reason for this.  One 
issue to consider is that depending on how the configure script is set 
up or structured, a test might not run at all.  But for example, if you 
have a check for a declaration of a function, and the test doesn't run 
in a particular configuration, the fallback in your own code would 
normally be to then manually declare the function yourself.  But if you 
didn't even run the test, then adding a declaration of a function you 
didn't want in the first place might be bad.  In that case, you can 
check with #ifdef whether the test was run, and then check the value of 
the macro for the test outcome.

On 2/7/22 21:40, Tom Lane wrote:
> I wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> What is the reason behind subtracting ccdlflags?
>> It looks like the coding actually originated here:
>> commit f5d0c6cad5bb2706e0e63f3f8f32e431ea428100
> Ah, here's the thread leading up to that:
>
> https://www.postgresql.org/message-id/flat/200106191206.f5JC6R108371%40candle.pha.pa.us
>
> The use of ldopts rather than hand-hacked link options seems to date to
> 0ed7864d6, only a couple days before that.  I don't think we had a
> buildfarm then, but I'd bet against the problem being especially
> widespread even then, or more people would've complained.


The buildfarm's first entry is from 22 Oct 2004.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andres Freund <andres@anarazel.de> writes:
> On 2022-02-07 20:42:09 -0500, Tom Lane wrote:
>> ... Peter just copied the logic in 7662419f1.  Considering that
>> the point of 7662419f1 was to get rid of MakeMaker, maybe we no
>> longer needed that at that point.

> Yea. And maybe what was broken in 2001 isn't broken anymore either ;)

Yeah --- note that Bruce was complaining about a problem on
Perl 5.005, which was already a bit over-the-hill in 2001.

> AIX is the one exception. Specifying -bE... doesn't seem right for building
> plperl etc. So possibly the subtraction accidentally does work for us there...

I tried this on AIX 7.2 (using the gcc farm, same build options
as hoverfly).  The build still works and passes regression tests,
but you get a warning about each symbol exported by Perl itself:

...
ld: 0711-415 WARNING: Symbol PL_veto_cleanup is already exported.
ld: 0711-415 WARNING: Symbol PL_warn_nl is already exported.
ld: 0711-415 WARNING: Symbol PL_warn_nosemi is already exported.
ld: 0711-415 WARNING: Symbol PL_warn_reserved is already exported.
ld: 0711-415 WARNING: Symbol PL_warn_uninit is already exported.
ld: 0711-415 WARNING: Symbol PL_WB_invlist is already exported.
ld: 0711-415 WARNING: Symbol PL_XPosix_ptrs is already exported.
ld: 0711-415 WARNING: Symbol PL_Yes is already exported.
ld: 0711-415 WARNING: Symbol PL_Zero is already exported.

So there's about 1200 such warnings for plperl, and then the same
again for each contrib foo_plperl module.  Maybe that's annoying
enough that we should keep the logic.  OTOH, it seems entirely
accidental that it has that effect.  I'd be a little inclined to
replace it with some rule about stripping '-bE:' switches out of
the ldopts result.

            regards, tom lane

- PGXS - and I don't yet know what to best do about it. One

  backward-compatible way would be to continue use makefiles for pgxs,

  but do the necessary replacement of Makefile.global.in via meson (and

  not use that for postgres' own build).  But that doesn't really

  provide a nicer path for building postgres extensions on windows, so

  it'd definitely not be a long-term path.

Hi,

On 2022-02-08 18:42:33 -0500, Tom Lane wrote:
> I'd be a little inclined to replace it with some rule about stripping '-bE:'
> switches out of the ldopts result.

Similar. That's a lot easier to understand than than -bE ending up stripped by
what we're doing. Should I do so, or do you want to?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-02-08 18:42:33 -0500, Tom Lane wrote:
>> I'd be a little inclined to replace it with some rule about stripping '-bE:'
>> switches out of the ldopts result.

> Similar. That's a lot easier to understand than than -bE ending up stripped by
> what we're doing. Should I do so, or do you want to?

I could look at it later, but if you want to do it, feel free.

            regards, tom lane

On 2/6/22 15:57, Andrew Dunstan wrote:
> On 2/6/22 13:39, Andres Freund wrote:
>> Hi,
>>
>> On 2022-02-06 12:06:41 -0500, Andrew Dunstan wrote:
>>> Here's a patch. I've tested the perl piece on master and it works fine.
>>> It applies cleanly down to 9.4, which is before we got transform modules
>>> (9.5) which fail if we just omit doing this platform-specific piece.
>> Given https://postgr.es/m/34e972bc-6e75-0754-9e6d-cde2518773a1%40dunslane.net
>> wouldn't it make sense to simply remove the pexports/gendef logic instead of
>> moving to gendef?
>>
> I haven't found a way to fix the transform builds if we do that. So
> let's leave that as a separate exercise unless you have a solution for
> that - this patch is really trivial.
>
>

Any objection to my moving ahead with this? My current workaround is this:


cat > /usr/bin/pexports <<EOF
#!/bin/sh
/ucrt64/bin/gendef - "$@"
EOF
chmod +x /usr/bin/pexports


(gendef is available in the ucrt64/mingw-w64-ucrt-x86_64-tools-git
package on msys2)


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2022-02-10 12:00:16 -0500, Andrew Dunstan wrote:
> Any objection to my moving ahead with this?

No. I don't yet understand what the transforms issue is and whether it can be
avoidded, but clearly it's an improvement to be able to build with builtin
msys tools vs not...

Greetings,

Andres Freund

On 2/10/22 12:52, Andres Freund wrote:
> Hi,
>
> On 2022-02-10 12:00:16 -0500, Andrew Dunstan wrote:
>> Any objection to my moving ahead with this?
> No. I don't yet understand what the transforms issue is and whether it can be
> avoidded, but clearly it's an improvement to be able to build with builtin
> msys tools vs not...



OK, thanks, done.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Is there a current patch set to review in this thread at the moment?

Hi,

On 2022-03-07 14:56:24 +0100, Peter Eisentraut wrote:
> Is there a current patch set to review in this thread at the moment?

I've been regularly rebasing and improving the patchset, but didn't post to
the thread about it most of the time.

I've just pushed another rebase, will work to squash it into a reasonable
number of patches and then repost that.

Greetings,

Andres Freund

Hi,

Attached is v6 of the meson patchset. There are a lots of changes since the
last version posted. These include:
- python2 removal is now committed, so not needed in here anymore
- CI changed to be based on the CI now merged into postgres
- CI also tests suse, rhel, fedora (Nazir Bilal Yavuz). Found several bugs. I
  don't think we'd merge all of those, but while working on the meson branch,
  it's really useful.
- all dependencies, except for pl/tcl (should be done soon)
- several missing options added (segsize, extra_{lib,include}_dirs, enable-tap-tests
- several portability fixes, builds on net/openbsd without changes now
- improvements to a number of "configure" tests
- lots of ongoing rebasing changes
- ...

Greetings,

Andres Freund

On 2022-03-07 09:58:41 -0800, Andres Freund wrote:
> On 2022-03-07 14:56:24 +0100, Peter Eisentraut wrote:
> > Is there a current patch set to review in this thread at the moment?
> 
> I've been regularly rebasing and improving the patchset, but didn't post to
> the thread about it most of the time.
> 
> I've just pushed another rebase, will work to squash it into a reasonable
> number of patches and then repost that.

Now done, see https://postgr.es/m/20220308025629.3xh2yo4sau74oafo%40alap3.anarazel.de

On 08.03.22 03:56, Andres Freund wrote:
> Attached is v6 of the meson patchset. There are a lots of changes since the
> last version posted. These include:
> - python2 removal is now committed, so not needed in here anymore
> - CI changed to be based on the CI now merged into postgres
> - CI also tests suse, rhel, fedora (Nazir Bilal Yavuz). Found several bugs. I
>    don't think we'd merge all of those, but while working on the meson branch,
>    it's really useful.
> - all dependencies, except for pl/tcl (should be done soon)
> - several missing options added (segsize, extra_{lib,include}_dirs, enable-tap-tests
> - several portability fixes, builds on net/openbsd without changes now
> - improvements to a number of "configure" tests
> - lots of ongoing rebasing changes
> - ...

I looked at this today mainly to consider whether some of the prereq
work is ready for adoption now.  A lot of the work has to do with
making various scripts write the output to other directories.  I
suspect this has something to do with how meson handles separate build
directories and how we have so far handled files created in the
distribution tarball.  But the whole picture isn't clear to me.

More generally, I don't see a distprep target in the meson build
files.  I wonder what your plan for that is, or whether that would
even work under meson.  In [0], I argued for getting rid of the
distprep step.  Perhaps it is time to reconsider that now.

[0]: 
https://www.postgresql.org/message-id/flat/cf0bec33-d965-664d-e0ec-fb15290f2273%402ndquadrant.com

For the short term, I think the patches 0002, 0008, 0010, and 0011
could be adopted, if they are finished as described.

Patch 0007 seems unrelated, or at least independently significant, and
should be discussed separately.

The rest is really all part of the same
put-things-in-the-right-directory issue.

For the overall patch set, I did a quick test with

meson setup build
cd build
ninja

which failed with

Undefined symbols for architecture x86_64:
   "_bbsink_zstd_new", referenced from:
       _SendBaseBackup in replication_basebackup.c.o

So maybe your patch set is not up to date with this new zstd build
option.


Details:

v6-0001-meson-prereq-output-and-depencency-tracking-work.patch.gz

This all looks kind of reasonable, but lacks explanation in some
cases, so I can't fully judge it.

v6-0002-meson-prereq-move-snowball_create.sql-creation-in.patch.gz

Looks like a reasonable direction, would be good to deduplicate with
Install.pm.

v6-0003-meson-prereq-add-output-path-arg-in-generate-lwlo.patch.gz

Ok.  Similar to 0001.  (But unlike 0001, nothing in this patch
actually uses the new output dir option.  That only comes in 0013.)

v6-0004-meson-prereq-add-src-tools-gen_versioning_script..patch.gz

This isn't used until 0013, and there it is patched again, so I'm not
sure if this is in the right position of this patch series.

v6-0005-meson-prereq-generate-errcodes.pl-accept-output-f.patch.gz

Also similar to 0001.

v6-0006-meson-prereq-remove-unhelpful-chattiness-in-snowb.patch.gz

Might as well include this into 0002.

v6-0007-meson-prereq-Can-we-get-away-with-not-export-all-.patch.gz

This is a separate discussion.  It's not clear to me why this is part
of this patch series.

v6-0008-meson-prereq-Handle-DLSUFFIX-in-msvc-builds-simil.patch.gz

Part of this was already done in 0001, so check if these patches are
split correctly.

I think the right way here is actually to go the other way around:
Move DLSUFFIX into header files for all platforms.  Move the DLSUFFIX
assignment from src/makefiles/ to src/templates/, have configure read
it, and then substitute it into Makefile.global and pg_config.h.

Then we also don't have to patch the Windows build code a bunch of
times to add the DLSUFFIX define everywhere.

There is code in configure already that would benefit from this, which
currently says

# We don't know the platform DLSUFFIX here, so check 'em all.

v6-0009-prereq-make-unicode-targets-work-in-vpath-builds.patch.gz

Another directory issue

v6-0010-ldap-tests-Don-t-run-on-unsupported-operating-sys.patch.gz

Not sure what this is supposed to do, but it looks independent of this
patch series.  Does it currently not work on "unsupported" operating
systems?

v6-0011-ldap-tests-Add-paths-for-openbsd.patch.gz

The more the merrier, although I'm a little bit worried about pointing
to a /usr/local/share/examples/ directory.

v6-0012-wip-split-TESTDIR-into-two.patch.gz
v6-0013-meson-Add-meson-based-buildsystem.patch.gz
v6-0014-meson-ci-Build-both-with-meson-and-as-before.patch.gz

I suggest in the interim to add a README.meson to show how to invoke
this.  Eventually, of course, we'd rewrite the installation
instructions.

Hi,

On 2022-03-09 13:37:23 +0100, Peter Eisentraut wrote:
> I looked at this today mainly to consider whether some of the prereq
> work is ready for adoption now.

Thanks!


> A lot of the work has to do with
> making various scripts write the output to other directories.  I
> suspect this has something to do with how meson handles separate build
> directories and how we have so far handled files created in the
> distribution tarball.  But the whole picture isn't clear to me.

A big part of it is that when building with ninja tools are invoked in the
top-level build directory, but right now a bunch of our scripts put their
output in CWD.


> More generally, I don't see a distprep target in the meson build
> files.  I wonder what your plan for that is, or whether that would
> even work under meson.  In [0], I argued for getting rid of the
> distprep step.  Perhaps it is time to reconsider that now.
> 
> [0]: https://www.postgresql.org/message-id/flat/cf0bec33-d965-664d-e0ec-fb15290f2273%402ndquadrant.com

I think it should be doable to add something roughly like the current distprep. The
cleanest way would be to use
https://mesonbuild.com/Reference-manual_builtin_meson.html#mesonadd_dist_script
to copy the files into the generated tarball.

Of course not adding it would be even easier ;)


> For the short term, I think the patches 0002, 0008, 0010, and 0011
> could be adopted, if they are finished as described.

Cool.


> Patch 0007 seems unrelated, or at least independently significant, and
> should be discussed separately.

It's related - it saves us from doing a lot of extra complexity on
windows. I've brought it up as a separate thread too:
https://postgr.es/m/20211101020311.av6hphdl6xbjbuif%40alap3.anarazel.de

I'm currently a bit stuck implementing this properly for the configure / make
system, as outlined in:
https://www.postgresql.org/message-id/20220111025328.iq5g6uck53j5qtin%40alap3.anarazel.de


> The rest is really all part of the same put-things-in-the-right-directory
> issue.
> 
> For the overall patch set, I did a quick test with
> 
> meson setup build
> cd build
> ninja
> 
> which failed with
> 
> Undefined symbols for architecture x86_64:
>   "_bbsink_zstd_new", referenced from:
>       _SendBaseBackup in replication_basebackup.c.o
> 
> So maybe your patch set is not up to date with this new zstd build
> option.

Yep, I posted it before "7cf085f077d - Add support for zstd base backup
compression."  went in, but after 6c417bbcc8f. So the meson build knew about
the zstd dependency, but didn't yet specify it for postgres /
pg_basebackup. So all that's needed was / is adding the dependency to those
two places.

Updated patches attached. This just contains the fix for this issue, doesn't
yet address review comments.

FWIW, I'd already pushed those fixes out to the git tree. There's frequent
enough small changes that reposting everytime seems too noisy.


> v6-0001-meson-prereq-output-and-depencency-tracking-work.patch.gz
> 
> This all looks kind of reasonable, but lacks explanation in some
> cases, so I can't fully judge it.

I'll try to clean it up.


> v6-0007-meson-prereq-Can-we-get-away-with-not-export-all-.patch.gz
> 
> This is a separate discussion.  It's not clear to me why this is part
> of this patch series.

See above.


> v6-0008-meson-prereq-Handle-DLSUFFIX-in-msvc-builds-simil.patch.gz
> 
> Part of this was already done in 0001, so check if these patches are
> split correctly.
> 
> I think the right way here is actually to go the other way around:
> Move DLSUFFIX into header files for all platforms.  Move the DLSUFFIX
> assignment from src/makefiles/ to src/templates/, have configure read
> it, and then substitute it into Makefile.global and pg_config.h.
> 
> Then we also don't have to patch the Windows build code a bunch of
> times to add the DLSUFFIX define everywhere.
> 
> There is code in configure already that would benefit from this, which
> currently says
> 
> # We don't know the platform DLSUFFIX here, so check 'em all.

I'll try it out.


> v6-0009-prereq-make-unicode-targets-work-in-vpath-builds.patch.gz
> 
> Another directory issue

I think it's a tad different, in that it's fixing something that's currently
broken in VPATH builds.


> v6-0010-ldap-tests-Don-t-run-on-unsupported-operating-sys.patch.gz
> 
> Not sure what this is supposed to do, but it looks independent of this
> patch series.  Does it currently not work on "unsupported" operating
> systems?

Right now if you run the ldap tests on windows, openbsd, ... the tests
fail. The only reason it doesn't cause trouble on the buildfarm is that we
currently don't run those tests by default...


> v6-0011-ldap-tests-Add-paths-for-openbsd.patch.gz
> 
> The more the merrier, although I'm a little bit worried about pointing
> to a /usr/local/share/examples/ directory.

It's where the files are in the package :/.


> v6-0012-wip-split-TESTDIR-into-two.patch.gz
> v6-0013-meson-Add-meson-based-buildsystem.patch.gz
> v6-0014-meson-ci-Build-both-with-meson-and-as-before.patch.gz
> 
> I suggest in the interim to add a README.meson to show how to invoke
> this.  Eventually, of course, we'd rewrite the installation
> instructions.

Good idea.


Greetings,

Andres Freund

Hi,

One thing that's pretty cool with ninja based builds is that it contains a
dependency log of "discovered" dependencies as well as information about
dependencies "encoded" in the build specification. LLVM contains a script that
uses that dependency information to see whether the build specification is
missing any dependencies - this helped me find several "build bugs".

The script is at:
https://github.com/llvm/llvm-project/blob/main/llvm/utils/check_ninja_deps.py

It just needs to be invoked in the build directory after a build.

If I e.g. remove the dependency from libpgcommon_srv.a to the lwlocknames.h
generation (*) it complains with:
error: src/common/libpgcommon_srv.a.p/cryptohash_openssl.c.o requires src/include/storage/lwlocknames.h but has no
dependencyon it
 
error: src/common/libpgcommon_srv.a.p/hmac_openssl.c.o requires src/include/storage/lwlocknames.h but has no dependency
onit
 


I wonder if it's worth having a build target invoking it? But how to get the
path to the script? We could just copy it into src/tools? It's permissively
licensed...

Greetings,

Andres Freund


(*) It seems architecturally pretty darn ugly for pgcommon to acquire lwlocks.

Hi,

On 2021-10-12 02:08:29 -0700, Andres Freund wrote:
> A very helpful visualization is to transform ninja's build logs into a
> tracefile with https://github.com/nico/ninjatracing
> 
> I attached an example - the trace.json.gz can be uploaded as-is to
> https://ui.perfetto.dev/

These days perfetto can load .ninja_log directly without conversion. Attached
is an example of the output.  Also attached a local .ninja_log to upload if
somebody wants to look at the interactive output without building.


> It's quite a bit of of fun to look at imo.
> 
> There's a few other things quickly apparent:
> 
> - genbki prevents build progress due to dependencies on the generated
>   headers.

That's still the biggest "slowdown".  Only a small number of things can start
before genbki is done. Parts of pgport, bison/flex and parts of the docs
build.


> - the absolutely stupid way I implemented the python2->python3
>   regression test output conversion uses up a fair bit of resources

That's gone now.


> - tablecmds.c, pg_dump.c, xlog.c and a few other files are starting to
>   big enough to be problematic compile-time wise

But these are still present. When building just the backend, the build speed
is limited by gram.y->gram.c, gram.c -> gram.o, linking postgres.

Greetings,

Andres Freund

Hi,

On 2021-10-22 11:55:05 -0400, John Naylor wrote:
> On Thu, Oct 21, 2021 at 5:48 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > However, update-unicode is a bit harder.  Partially not directly because
> of
> > meson, but because update-unicode as-is afaict doesn't support VPATH
> builds,
> > and meson enforces those.
> 
> > make update-unicode
> > ...
> > make -C src/common/unicode update-unicode
> > '/usr/bin/perl' generate-unicode_norm_table.pl
> > Can't open perl script "generate-unicode_norm_table.pl": No such file or
> directory
> >
> > It's not too hard to fix. See attached for the minimal stuff that I
> > immediately found to be needed.
> 
> Thanks for doing that, it works well enough for demonstration. With your
> patch, and using an autoconf VPATH build, the unicode tables work fine, but
> it complains of a permission error in generate_unaccent_rules.py. That
> seems to be because the script is invoked directly rather than as an
> argument to the python interpreter.
>

> Yeah. I encountered a further issue: With autoconf on HEAD, with a source
> tree build executed in contrib/unaccent:

This seems to be the same issue as above?


> $ touch generate_unaccent_rules.py
> $ make update-unicode
> generate_unaccent_rules.py --unicode-data-file
> ../../src/common/unicode/UnicodeData.txt --latin-ascii-file Latin-ASCII.xml
> >unaccent.rules
> /bin/sh: generate_unaccent_rules.py: command not found
> make: *** [unaccent.rules] Error 127
> make: *** Deleting file `unaccent.rules'

This looks more like you're building without --with-python and you don't have
a 'python' binary (but a python3 binary)?

Independent of my changes the invocation of generate_unaccent_rules looks like

# Allow running this even without --with-python
PYTHON ?= python
...
    $(PYTHON) $< --unicode-data-file $(word 2,$^) --latin-ascii-file $(word 3,$^) >$@

so your failure should only happen if PYTHON somehow is empty, otherwise I'd
expect python in front of the failing line?


> Anyway, this can be put off until the very end, since it's not run often.
> You've demonstrated how these targets would work, and that's good enough
> for now.

I'd like to get this stuff out of the patch series, so I'm planning to get
this committable...

Greetings,

Andres Freund

On 09.03.22 17:44, Andres Freund wrote:
>> v6-0009-prereq-make-unicode-targets-work-in-vpath-builds.patch.gz
>>
>> Another directory issue
> I think it's a tad different, in that it's fixing something that's currently
> broken in VPATH builds.

Ok, I took another look at this.

-override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+override CPPFLAGS := -DFRONTEND 
-I$(abs_top_builddir)/src/common/unicode $(CPPFLAGS)

This could just be

-I.

-   $(PERL) generate-unicode_norm_table.pl
+   $(PERL) $< $(CURDIR)

I didn't detect a need for the additional directory argument.  (So the 
changes in generate-unicode_norm_table.pl are also apparently not 
necessary.)  Maybe this is something that will become useful later, in 
which case it should be split out from this patch.

The rest of this patch looks ok.

Hi,

Attached is v8. It's just a rebase to resolve conflicts with recent changes.

- Andres

On 09.03.22 13:37, Peter Eisentraut wrote:
> v6-0008-meson-prereq-Handle-DLSUFFIX-in-msvc-builds-simil.patch.gz
> 
> I think the right way here is actually to go the other way around:
> Move DLSUFFIX into header files for all platforms.  Move the DLSUFFIX
> assignment from src/makefiles/ to src/templates/, have configure read
> it, and then substitute it into Makefile.global and pg_config.h.
> 
> Then we also don't have to patch the Windows build code a bunch of
> times to add the DLSUFFIX define everywhere.

This patch should do it.

Hi,

On 2022-03-24 16:16:15 +0100, Peter Eisentraut wrote:
> On 09.03.22 13:37, Peter Eisentraut wrote:
> > v6-0008-meson-prereq-Handle-DLSUFFIX-in-msvc-builds-simil.patch.gz
> > 
> > I think the right way here is actually to go the other way around:
> > Move DLSUFFIX into header files for all platforms.  Move the DLSUFFIX
> > assignment from src/makefiles/ to src/templates/, have configure read
> > it, and then substitute it into Makefile.global and pg_config.h.
> > 
> > Then we also don't have to patch the Windows build code a bunch of
> > times to add the DLSUFFIX define everywhere.
> 
> This patch should do it.

> From 763943176a1e0a0c954414ba9da07742ad791656 Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Thu, 24 Mar 2022 16:00:54 +0100
> Subject: [PATCH] Refactor DLSUFFIX handling
> 
> Move DLSUFFIX into header files for all platforms.  Move the DLSUFFIX
> assignment from src/makefiles/ to src/templates/, have configure read
> it, and then substitute it into Makefile.global and pg_config.h.  This
> avoids the need of all users to locally set CPPFLAGS.

Reading through it, this looks good to me. Thanks!

Greetings,

Andres Freund

On 22.03.22 03:22, Andres Freund wrote:
> Attached is v8. It's just a rebase to resolve conflicts with recent changes.

I have committed the DLSUFFIX refactoring, and also a stripped-down 
version of the patch that makes update-unicode work with vpath.  This 
takes care of patches 0007 and 0009.

Patch 0006 (visibility) has its own CF entry.

The only other thing IMO that might be worth addressing in PG15 is the 
snowball scripts refactoring (0002 and 0003), but that doesn't seem 
quite ready yet.  (At least, it would need to be integrated into the 
distprep target, since it adds a dependency on perl.)  Maybe it's not 
worth it right now.

With that, I suggest moving this patch set to CF 2022-07.

A general comment on the remaining prereq patches:  We appear to be 
accumulating a mix of conventions for how "generate" scripts specify 
their output file.  Some have it built-in, some use the last argument, 
some use an option, which might be -o or --output.  Maybe we can gently 
work toward more commonality there.

Hi,

On 2022-03-25 10:01:09 +0100, Peter Eisentraut wrote:
> On 22.03.22 03:22, Andres Freund wrote:
> > Attached is v8. It's just a rebase to resolve conflicts with recent changes.
> 
> I have committed the DLSUFFIX refactoring, and also a stripped-down version
> of the patch that makes update-unicode work with vpath.  This takes care of
> patches 0007 and 0009.

Thanks!


> The only other thing IMO that might be worth addressing in PG15 is the
> snowball scripts refactoring (0002 and 0003), but that doesn't seem quite
> ready yet.  (At least, it would need to be integrated into the distprep
> target, since it adds a dependency on perl.)  Maybe it's not worth it right
> now.

Not sure myself.


> With that, I suggest moving this patch set to CF 2022-07.

Done.


One thing I'd like to discuss fairly soon is what kind of approach to take for
integrating meson support. Most other projects I looked kept parallel
buildsystems for at least a release, so that there's one round of "broad" user
feedback.

In our context it could make sense to merge meson, after a few months of
shakeup remove the current windows buildsystems, and then in release + 1
remove the make based stuff.

But we can have that discussion that before the next CF, but still after
code-freeze & immediate mopup.


> A general comment on the remaining prereq patches:  We appear to be
> accumulating a mix of conventions for how "generate" scripts specify their
> output file.  Some have it built-in, some use the last argument, some use an
> option, which might be -o or --output.  Maybe we can gently work toward more
> commonality there.

Fair point.

Greetings,

Andres Freund

On 3/28/22 15:59, Andres Freund wrote:
>
> One thing I'd like to discuss fairly soon is what kind of approach to take for
> integrating meson support. Most other projects I looked kept parallel
> buildsystems for at least a release, so that there's one round of "broad" user
> feedback.



We did something similar when we moved from CVS to git.


>
> In our context it could make sense to merge meson, after a few months of
> shakeup remove the current windows buildsystems, and then in release + 1
> remove the make based stuff.
>
> But we can have that discussion that before the next CF, but still after
> code-freeze & immediate mopup.
>


I'd like to get a stake in the ground and then start working on
buildfarm support. Up to now I think it's been a bit too much of a
moving target. Essentially that would mean an interim option for
building with meson.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:
> On 3/28/22 15:59, Andres Freund wrote:
>> In our context it could make sense to merge meson, after a few months of
>> shakeup remove the current windows buildsystems, and then in release + 1
>> remove the make based stuff.

That sounds like a decent plan.

> I'd like to get a stake in the ground and then start working on
> buildfarm support. Up to now I think it's been a bit too much of a
> moving target. Essentially that would mean an interim option for
> building with meson.

If we can commit meson build infrastructure without removing the
existing infrastructure, then the buildfarm can continue to work,
and we can roll out support for the new way slowly.  It'd be
fairly impractical to expect all buildfarm animals to update
instantly anyhow.

            regards, tom lane

Hi,

On 2022-03-28 18:58:19 -0400, Tom Lane wrote:
> If we can commit meson build infrastructure without removing the
> existing infrastructure, then the buildfarm can continue to work,
> and we can roll out support for the new way slowly.

I think it's not a huge issue to have both for a while. Of course it's
annoying to have to update two files when adding a source file, but it's not
the end of the world for a limited time. IMO.


> It'd be fairly impractical to expect all buildfarm animals to update
> instantly anyhow.

And development workflows. I expect some unforseen breakages somewhere, given
the variety of systems out there.

Greetings,

Andres Freund

Some feedback and patches for your branch at 
3274198960c139328fef3c725cee1468bbfff469:

0001-Install-a-few-more-files.patch

These are just some files that were apparently forgotten to be installed 
so far.

0002-Adjust-some-header-file-installation-paths.patch

The installation of server headers is apparently still in progress. 
This just adjusts the installation directory of those that are already 
being dealt with, so they match the existing installation layout.

0003-Fix-warnings-about-deprecated-features.patch

This fixes some deprecation warnings and raises the requirement to 0.56. 
  I'm not sure why the current cutoff at 0.54 was chosen.  Perhaps that 
could be documented.  If we choose to stay with 0.54, is there a way to 
turn off deprecation warnings, so not everyone needs to see them?

0004-Install-postmaster-symlink.patch

This needs 0.61, so maybe it's a bit too new.  Or we could get rid of 
the postmaster symlink altogether?

0005-Workaround-for-Perl-detection.patch

This is needed on my system to get the Perl detection to pass.  If I 
look at the equivalent configure code, some more refinement appears to 
be needed in this area.

On 13.04.22 12:26, Peter Eisentraut wrote:
> Some feedback and patches for your branch at 
> 3274198960c139328fef3c725cee1468bbfff469:

Here is another patch.  It adds support for building ecpg.

Hi,

On 2022-04-13 12:26:05 +0200, Peter Eisentraut wrote:
> Some feedback and patches for your branch at
> 3274198960c139328fef3c725cee1468bbfff469:

Thanks! I just rebased the branch, will merge your changes once the fallout
from that is fixed...


> 0001-Install-a-few-more-files.patch
> 
> These are just some files that were apparently forgotten to be installed so
> far.

> 0002-Adjust-some-header-file-installation-paths.patch
> 
> The installation of server headers is apparently still in progress. This
> just adjusts the installation directory of those that are already being
> dealt with, so they match the existing installation layout.


Yea. I've not at all paid attention to that so far, besides getting tests to
pass.


> 0003-Fix-warnings-about-deprecated-features.patch
> 
> This fixes some deprecation warnings and raises the requirement to 0.56.

I don't see any deprecation warnings - I see some notices about *future*
deprecated features being used:

NOTICE: Future-deprecated features used:
 * 0.55.0: {'ExternalProgram.path'}
 * 0.56.0: {'meson.source_root', 'meson.build_root'}

(i.e. once the minimum version is increased to > 0.54, those will trigger
deprecation warnings)

What are you seeing with what version?


> I'm not sure why the current cutoff at 0.54 was chosen.  Perhaps that could
> be documented.

Not quite sure why I ended up with 0.54. We definitely should require at most
0.56, as that's the last version supporting python 3.5.


> 0004-Install-postmaster-symlink.patch
> 
> This needs 0.61, so maybe it's a bit too new.

Yea, that's too new. I think we can just create the symlink using ln or such
if we need it.


> Or we could get rid of the postmaster symlink altogether?

But that seems like a better approach.


> 0005-Workaround-for-Perl-detection.patch
> 
> This is needed on my system to get the Perl detection to pass.  If I look at
> the equivalent configure code, some more refinement appears to be needed in
> this area.

> From 1f80e1ebb8efeb0eba7d57032282520fd6455b0d Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Wed, 13 Apr 2022 11:50:52 +0200
> Subject: [PATCH 5/5] Workaround for Perl detection
> 
> ---
>  meson.build | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meson.build b/meson.build
> index 1bf53ea24d..e33ed11b08 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -545,9 +545,9 @@ else
>    # file existence.
>    if perl_may_work
>      perl_ccflags += ['-I@0@'.format(perl_inc_dir)]
> -    if host_machine.system() == 'darwin'
> -      perl_ccflags += ['-iwithsysroot', perl_inc_dir]
> -    endif
> +    #if host_machine.system() == 'darwin'
> +    #  perl_ccflags += ['-iwithsysroot', perl_inc_dir]
> +    #endif
>    endif

What problem do you see without this? It did build on CI and on my m1 mini box
as is...

Greetings,

Andres Freund

On 20.04.22 23:04, Andres Freund wrote:
>> 0003-Fix-warnings-about-deprecated-features.patch
>>
>> This fixes some deprecation warnings and raises the requirement to 0.56.
> 
> I don't see any deprecation warnings - I see some notices about *future*
> deprecated features being used:
> 
> NOTICE: Future-deprecated features used:
>   * 0.55.0: {'ExternalProgram.path'}
>   * 0.56.0: {'meson.source_root', 'meson.build_root'}
> 
> (i.e. once the minimum version is increased to > 0.54, those will trigger
> deprecation warnings)
> 
> What are you seeing with what version?

I see the same thing.  Effectively, "deprecation warning" and 
"future-deprecation notice" are just different spellings of "yelling at 
me unconditionally for using code that I can't do anything about".

>> I'm not sure why the current cutoff at 0.54 was chosen.  Perhaps that could
>> be documented.
> 
> Not quite sure why I ended up with 0.54. We definitely should require at most
> 0.56, as that's the last version supporting python 3.5.

Why is Python 3.5 relevant?

>>  From 1f80e1ebb8efeb0eba7d57032282520fd6455b0d Mon Sep 17 00:00:00 2001
>> From: Peter Eisentraut <peter@eisentraut.org>
>> Date: Wed, 13 Apr 2022 11:50:52 +0200
>> Subject: [PATCH 5/5] Workaround for Perl detection
>>
>> ---
>>   meson.build | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meson.build b/meson.build
>> index 1bf53ea24d..e33ed11b08 100644
>> --- a/meson.build
>> +++ b/meson.build
>> @@ -545,9 +545,9 @@ else
>>     # file existence.
>>     if perl_may_work
>>       perl_ccflags += ['-I@0@'.format(perl_inc_dir)]
>> -    if host_machine.system() == 'darwin'
>> -      perl_ccflags += ['-iwithsysroot', perl_inc_dir]
>> -    endif
>> +    #if host_machine.system() == 'darwin'
>> +    #  perl_ccflags += ['-iwithsysroot', perl_inc_dir]
>> +    #endif
>>     endif
> 
> What problem do you see without this? It did build on CI and on my m1 mini box
> as is...

I'm using homebrew-installed gcc and homebrew-installed perl.  gcc 
doesn't understand the option -iwithsysroot, and apparently whatever it 
points to is not needed.

Note that in configure.ac the logic is like this:

   if test \! -f "$perl_archlibexp/CORE/perl.h" ; then
     if test -f "$PG_SYSROOT$perl_archlibexp/CORE/perl.h" ; then
       perl_includespec="-iwithsysroot $perl_archlibexp/CORE"
     fi
   fi

So it checks first if it can find the needed file without the sysroot 
business.

Hi,

On 2022-04-21 22:36:01 +0200, Peter Eisentraut wrote:
> On 20.04.22 23:04, Andres Freund wrote:
> > > 0003-Fix-warnings-about-deprecated-features.patch
> > > 
> > > This fixes some deprecation warnings and raises the requirement to 0.56.
> > 
> > I don't see any deprecation warnings - I see some notices about *future*
> > deprecated features being used:
> > 
> > NOTICE: Future-deprecated features used:
> >   * 0.55.0: {'ExternalProgram.path'}
> >   * 0.56.0: {'meson.source_root', 'meson.build_root'}
> > 
> > (i.e. once the minimum version is increased to > 0.54, those will trigger
> > deprecation warnings)
> > 
> > What are you seeing with what version?
> 
> I see the same thing.  Effectively, "deprecation warning" and
> "future-deprecation notice" are just different spellings of "yelling at me
> unconditionally for using code that I can't do anything about".

Yea, I'm not happy that "future-deprecation notice" was enabled by
default. It's still different from "deprecation warning" in prominence and
behaviour (e.g. --fatal-meson-warnings doesn't error out for notices but not
for errors), but ...

Might be worth raising with the meson folks.


> > > I'm not sure why the current cutoff at 0.54 was chosen.  Perhaps that could
> > > be documented.
> > 
> > Not quite sure why I ended up with 0.54. We definitely should require at most
> > 0.56, as that's the last version supporting python 3.5.
> 
> Why is Python 3.5 relevant?

It's the latest available on some older platforms. It's pretty easy to install
a new meson, a heck of a lot more work to install a new python. IIRC solaris,
AIX and some of Tom's dinosaurs.


> > >  From 1f80e1ebb8efeb0eba7d57032282520fd6455b0d Mon Sep 17 00:00:00 2001
> > > From: Peter Eisentraut <peter@eisentraut.org>
> > > Date: Wed, 13 Apr 2022 11:50:52 +0200
> > > Subject: [PATCH 5/5] Workaround for Perl detection
> > > 
> > > ---
> > >   meson.build | 6 +++---
> > >   1 file changed, 3 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/meson.build b/meson.build
> > > index 1bf53ea24d..e33ed11b08 100644
> > > --- a/meson.build
> > > +++ b/meson.build
> > > @@ -545,9 +545,9 @@ else
> > >     # file existence.
> > >     if perl_may_work
> > >       perl_ccflags += ['-I@0@'.format(perl_inc_dir)]
> > > -    if host_machine.system() == 'darwin'
> > > -      perl_ccflags += ['-iwithsysroot', perl_inc_dir]
> > > -    endif
> > > +    #if host_machine.system() == 'darwin'
> > > +    #  perl_ccflags += ['-iwithsysroot', perl_inc_dir]
> > > +    #endif
> > >     endif
> > 
> > What problem do you see without this? It did build on CI and on my m1 mini box
> > as is...
> 
> I'm using homebrew-installed gcc and homebrew-installed perl.  gcc doesn't
> understand the option -iwithsysroot, and apparently whatever it points to is
> not needed.

Ah, I only tested with system "cc".


> Note that in configure.ac the logic is like this:
> 
>   if test \! -f "$perl_archlibexp/CORE/perl.h" ; then
>     if test -f "$PG_SYSROOT$perl_archlibexp/CORE/perl.h" ; then
>       perl_includespec="-iwithsysroot $perl_archlibexp/CORE"
>     fi
>   fi
> 
> So it checks first if it can find the needed file without the sysroot
> business.

I guess we'll have to copy that. Although it doesn't seem quite the right
behaviour, because it might end up picking up a different perl.h that way...

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-04-21 22:36:01 +0200, Peter Eisentraut wrote:
>> Why is Python 3.5 relevant?

> It's the latest available on some older platforms. It's pretty easy to install
> a new meson, a heck of a lot more work to install a new python. IIRC solaris,
> AIX and some of Tom's dinosaurs.

FWIW, I don't think that either gaur or prairiedog need be factored into
this conversation.  They cannot build ninja at all for lack of <spawn.h>,
so whether they could run meson is pretty much beside the point.

(I wonder if we should stick in a configure test for <spawn.h>,
just to see if anything else doesn't have it?)

We should worry a little more about Solaris and AIX, but even there I
think it's largely up to the platform owner whether they've updated
python to something modern.  If it isn't, you need to move the goalposts
back some more :-(.  As of today I see the following pre-3.6 pythons
in the buildfarm, exclusive of mine:

skate        3.2.3
snapper        3.2.3
topminnow    3.4.2
hornet        3.4.3
sungazer    3.4.3
wrasse        3.4.3
shelduck    3.4.10
curculio    3.5.1
hoverfly    3.5.1
batfish        3.5.2
spurfowl    3.5.2
cuon        3.5.2
ayu        3.5.3
chimaera    3.5.3
chipmunk    3.5.3
grison        3.5.3
mussurana    3.5.3
tadarida    3.5.3
urocryon    3.5.3

            regards, tom lane

On 2022-04-21 Th 17:34, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On 2022-04-21 22:36:01 +0200, Peter Eisentraut wrote:
>>> Why is Python 3.5 relevant?
>> It's the latest available on some older platforms. It's pretty easy to install
>> a new meson, a heck of a lot more work to install a new python. IIRC solaris,
>> AIX and some of Tom's dinosaurs.
> FWIW, I don't think that either gaur or prairiedog need be factored into
> this conversation.  They cannot build ninja at all for lack of <spawn.h>,
> so whether they could run meson is pretty much beside the point.
>
> (I wonder if we should stick in a configure test for <spawn.h>,
> just to see if anything else doesn't have it?)
>
> We should worry a little more about Solaris and AIX, but even there I
> think it's largely up to the platform owner whether they've updated
> python to something modern.  If it isn't, you need to move the goalposts
> back some more :-(.  As of today I see the following pre-3.6 pythons
> in the buildfarm, exclusive of mine:
>
> skate        3.2.3
> snapper        3.2.3
> topminnow    3.4.2
> hornet        3.4.3
> sungazer    3.4.3
> wrasse        3.4.3
> shelduck    3.4.10
> curculio    3.5.1
> hoverfly    3.5.1
> batfish        3.5.2
> spurfowl    3.5.2
> cuon        3.5.2
> ayu        3.5.3
> chimaera    3.5.3
> chipmunk    3.5.3
> grison        3.5.3
> mussurana    3.5.3
> tadarida    3.5.3
> urocryon    3.5.3
>
>             


Presumably that only tells you about the animals currently building with
python.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Here is a patch that adds in NLS.

There are some opportunities to improve this.  For example, we could 
move the list of languages from the meson.build files into separate 
LINGUAS files, which could be shared with the makefile-based build 
system.  I need to research this a bit more.

Also, this only covers the build and install phases of the NLS process. 
The xgettext and msgmerge aspects I haven't touched at all.  There is 
more to research there as well.

The annoying thing is that the i18n module doesn't appear to have a way 
to communicate with feature options or dependencies, so there isn't a 
way to tell it to only do its things when some option is enabled, or 
conversely to check whether the module found the things it needs and to 
enable or disable an option based on that.  So right now for example if 
you explicitly disable the 'nls' option, the binaries are built without 
NLS but the .mo files are still built and installed.

In any case, this works for the main use cases and gets us a step 
forward, so it's worth considering.

Hi,

On 2022-04-20 15:09:31 +0200, Peter Eisentraut wrote:
> On 13.04.22 12:26, Peter Eisentraut wrote:
> > Some feedback and patches for your branch at
> > 3274198960c139328fef3c725cee1468bbfff469:
> 
> Here is another patch.  It adds support for building ecpg.

Cool!

I have merged this, with a few changes (split parse.pl change out, changed its
invocation in Solution.pm, indentation, explicitly using shared_library()
rather than library(), indentation).

But there's need for some more - exports.txt handling is needed for windows
(and everywhere else, but not as crucially) - hence CI currently being broken
on windows. I've done that in a VM, and it indeed fixes the issues. But it
needs to be generalized, I just copied and pasted stuff around...

Greetings,

Andres Freund

Hi,

On 2022-04-27 21:56:27 +0200, Peter Eisentraut wrote:
> Here is a patch that adds in NLS.

Cool! I know very little about translations, so I was reticent tackling
this...


> For example, we could move the list of languages from the meson.build files
> into separate LINGUAS files, which could be shared with the makefile-based
> build system.  I need to research this a bit more.

Yea, that'd be nice.


> The annoying thing is that the i18n module doesn't appear to have a way to
> communicate with feature options or dependencies, so there isn't a way to
> tell it to only do its things when some option is enabled, or conversely to
> check whether the module found the things it needs and to enable or disable
> an option based on that.  So right now for example if you explicitly disable
> the 'nls' option, the binaries are built without NLS but the .mo files are
> still built and installed.

One partial way to deal with that, I think, would be to change all the
subdir('po') invocations to subdir('po', if_found: libintl). If we don't want
that for some reason, is there a reason a simple if libintl.found() wouldn't
work?


> In any case, this works for the main use cases and gets us a step forward,
> so it's worth considering.

Agreed.

Greetings,

Andres Freund

Hi,

On 2022-04-29 11:00:43 -0700, Andres Freund wrote:
> On 2022-04-27 21:56:27 +0200, Peter Eisentraut wrote:
> > Here is a patch that adds in NLS.
> 
> Cool! I know very little about translations, so I was reticent tackling
> this...

> > The annoying thing is that the i18n module doesn't appear to have a way to
> > communicate with feature options or dependencies, so there isn't a way to
> > tell it to only do its things when some option is enabled, or conversely to
> > check whether the module found the things it needs and to enable or disable
> > an option based on that.  So right now for example if you explicitly disable
> > the 'nls' option, the binaries are built without NLS but the .mo files are
> > still built and installed.
> 
> One partial way to deal with that, I think, would be to change all the
> subdir('po') invocations to subdir('po', if_found: libintl). If we don't want
> that for some reason, is there a reason a simple if libintl.found() wouldn't
> work?

Merged into my tree now, using if_found. I've also made the intl check work
with older meson versions, since I didn't include your version requirement
upgrades.


For now I "fixed" the ecpg issue on windows by just not building ecpg
there. Ecpg also needs tests ported...

Greetings,

Andres Freund

On 29.04.22 19:46, Andres Freund wrote:
> explicitly using shared_library() rather than library()

Why is that?  We do build static libraries right now, so using library() 
would seem more suitable for that.

Hi,

On 2022-05-02 16:47:43 +0200, Peter Eisentraut wrote:
> On 29.04.22 19:46, Andres Freund wrote:
> > explicitly using shared_library() rather than library()
> 
> Why is that?  We do build static libraries right now, so using library()
> would seem more suitable for that.

When I wrote this I hadn't realized that we build both shared and static
libraries. I've since changed the respective ecpg libraries to use
both_libraries(). Same with libpq (I really hadn't realized we build a static
libpq...).

Greetings,

Andres Freund

More patches:

0001-meson-Assorted-compiler-test-tweaks.patch

I was going through a diff of pg_config.h between old and new build and 
found a few omissions and small differences.

Some of the

     blah ? 1 : false

is of course annoying and can be removed eventually, but it's useful 
when analyzing the diff, and since it's already done in other places it 
seems reasonable to apply it consistently.

Of course there is some more work left for some of the more complicated 
tests; this isn't meant to be complete.


0002-meson-Add-pg_walinspect.patch

This was added more recently and was not ported yet.  Nothing too 
interesting here.


0003-meson-Install-all-server-headers.patch

With this, all the server headers installed by a makefile-based build 
are installed.  I tried to strike a balance between using 
install_subdir() with exclude list versus listing things explicitly. 
Different variations might be possible, but this looked pretty sensible 
to me.


With these patches, the list of files installed with make versus meson 
match up, except for known open items (postmaster symlink, some library 
naming differences, pkgconfig, pgxs, test modules installed, documentation).

Hi,

On 2022-05-04 13:53:54 +0200, Peter Eisentraut wrote:
> 0001-meson-Assorted-compiler-test-tweaks.patch
> 
> I was going through a diff of pg_config.h between old and new build and
> found a few omissions and small differences.

Thanks, merged that.


> is of course annoying and can be removed eventually, but it's useful when
> analyzing the diff, and since it's already done in other places it seems
> reasonable to apply it consistently.

Yea, I'd tried to minimize the difference at some point, but haven't done so
in a while...


> 0002-meson-Add-pg_walinspect.patch
> 
> This was added more recently and was not ported yet.  Nothing too
> interesting here.

Merged that.


> 0003-meson-Install-all-server-headers.patch
> 
> With this, all the server headers installed by a makefile-based build are
> installed.  I tried to strike a balance between using install_subdir() with
> exclude list versus listing things explicitly. Different variations might be
> possible, but this looked pretty sensible to me.

I locally had something similar, but I'm worried that this approach will be
too fragile. Leads to e.g. editor temp files getting installed. I've merged it
for now, but I think we need a different approach.


> With these patches, the list of files installed with make versus meson match
> up, except for known open items (postmaster symlink, some library naming
> differences, pkgconfig, pgxs, test modules installed, documentation).

I added pkgconfig since then. They're not exactly the same, but pretty close,
except for one thing: Looks like some of the ecpg libraries really should link
to some other ecpg libs? I think we're missing something there... That then
leads to missing requirements in the .pc files.

Re symlink: Do you have an opion about dropping the symlink vs implementing it
(likely via a small helper script?)?

Re library naming: It'd obviously be easy to adjust the library names, but I
wonder if it'd not be worth keeping the _static.a suffix, right now unsuffixed
library name imo is quite confusing.

Re test modules: Not sure what the best fix for that is yet. Except that we
don't have a search path for server libs, I'd just install them to a dedicated
path or add the build dir to the search path. But we don't, so ...

Re docs: I think the best approach here would be to have a new
meson_options.txt option defining whether the docs should be built. But not
quite sure.

Greetings,

Andres Freund

Hi,

On 2022-04-21 17:34:47 -0400, Tom Lane wrote:
> FWIW, I don't think that either gaur or prairiedog need be factored into
> this conversation.  They cannot build ninja at all for lack of <spawn.h>,
> so whether they could run meson is pretty much beside the point.

Yea.


> (I wonder if we should stick in a configure test for <spawn.h>,
> just to see if anything else doesn't have it?)

Might be worth doing?


> We should worry a little more about Solaris and AIX, but even there I
> think it's largely up to the platform owner whether they've updated
> python to something modern.

Looks like "AIX toolbox" is at 3.7. Solaris 11.4 apparently has 3.5 (11.3 is
EOL January 2024).

I think it's worth caring about supporting 3.6 due to RHEL 7 for now.


> If it isn't, you need to move the goalposts
> back some more :-(.  As of today I see the following pre-3.6 pythons
> in the buildfarm, exclusive of mine:
>
> skate        3.2.3
> snapper        3.2.3

Debian wheezy, I feel ok with dropping that.


> topminnow    3.4.2

Debian jessie, similar.


> hornet        3.4.3
> sungazer    3.4.3

Looks like a newer python version is available for AIX, without manually
compiling.


> wrasse        3.4.3

Apparently solaris 11.4 has python 3.5 (still not great :/)


> shelduck    3.4.10

This animal seems to have retired.


> curculio    3.5.1

Supported versions of openbsd have modern versions of python.


> hoverfly    3.5.1

AIX


> batfish        3.5.2
> spurfowl    3.5.2
> cuon        3.5.2

Ubuntu 16.04 is EOL (since 2021-04), outside of paid extended support.


> ayu        3.5.3
> chimaera    3.5.3
> chipmunk    3.5.3
> grison        3.5.3
> mussurana    3.5.3
> tadarida    3.5.3
> urocryon    3.5.3

These are all [variants of] debian stretch. I think we should be ok dropping
support for that, the extended "LTS" support for stretch ends June 30, 2022
(with the last non-extended update at July 18, 2020).

Greetings,

Andres Freund

[1] https://repology.org/project/python/versions

Hi,

On 2022-05-06 14:27:24 -0700, Andres Freund wrote:
> > 0003-meson-Install-all-server-headers.patch
> > 
> > With this, all the server headers installed by a makefile-based build are
> > installed.  I tried to strike a balance between using install_subdir() with
> > exclude list versus listing things explicitly. Different variations might be
> > possible, but this looked pretty sensible to me.
> 
> I locally had something similar, but I'm worried that this approach will be
> too fragile. Leads to e.g. editor temp files getting installed. I've merged it
> for now, but I think we need a different approach.

Meant to add potential alternatives here: The easiest likely would be to just
add an install script that globs *.h. Alternatively we could build a file list
at configure time, and then install that with install_header(). The advantage
would be that it be available for things like cpluspluscheck, the disadvantage
that something needs to trigger reconfiguration to update the file list.

Greetings,

Andres Freund

On 06.05.22 23:27, Andres Freund wrote:
> Re symlink: Do you have an opion about dropping the symlink vs implementing it
> (likely via a small helper script?)?

I think the postmaster symlink could be dropped.  The postmaster man 
page has been saying that it's deprecated since 2006.

More patches:

I fixed the Perl detection issue in my macOS environment that I had 
reported a while ago.

Then I added in support for all configure options that had not been 
ported over yet.  Some of these are rather trivial.

After that, these configure options don't have an equivalent yet:

--disable-rpath
--enable-profiling
--disable-thread-safety
--with-libedit-preferred

The first three overlap with meson built-in functionality, so we would 
need to check whether the desired functionality is available somehow.

The last one we probably want to keep somehow; it would need a bit of 
fiddly work.

Hi,

On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
> I fixed the Perl detection issue in my macOS environment that I had reported
> a while ago.

Hm. I wonder if it's right to check with is_file() - perhaps there are
platforms that have split off the include directory?


> Then I added in support for all configure options that had not been ported
> over yet.  Some of these are rather trivial.

Thanks!

Some of these (extra version, krb srvname, ...) I just merged from a
colleague.

Will look at merging the others.


> After that, these configure options don't have an equivalent yet:
> 
> --disable-rpath
> --enable-profiling
> --disable-thread-safety
> --with-libedit-preferred
> 
> The first three overlap with meson built-in functionality, so we would need
> to check whether the desired functionality is available somehow.

Which builtin functionality does --enable-profiling overlap with? There's a
coverage option, perhaps you were thinking of that?

I don't think we should add --disable-thread-safety, platforms without it also
aren't going to support ninja / meson... IIRC Thomas was planning to submit a
patch getting rid of it independently...


> The last one we probably want to keep somehow; it would need a bit of fiddly
> work.

A colleague just finished that bit. Probably can be improved further, but it
works now...


> From 049b34b6a8dd949f0eb7987cad65f6682a6ec786 Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Wed, 11 May 2022 09:06:13 +0200
> Subject: [PATCH 3/9] meson: prereq: Refactor dtrace postprocessing make rules
> 
> Move the dtrace postprocessing sed commands into a separate file so
> that it can be shared by meson.  Also split the rule into two for
> proper dependency declaration.

Hm. Using sed may be problematic on windows...


> From fad02f1fb71ec8c64e47e5031726ffbee4a1dd84 Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Wed, 11 May 2022 09:53:01 +0200
> Subject: [PATCH 7/9] meson: Add system-tzdata option
> 
> ---
>  meson.build       | 3 +++
>  meson_options.txt | 3 +++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/meson.build b/meson.build
> index 7c9c6e7f23..b33a51a35d 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -246,6 +246,9 @@ cdata.set('RELSEG_SIZE', get_option('segsize') * 131072)
>  cdata.set('DEF_PGPORT', get_option('pgport'))
>  cdata.set_quoted('DEF_PGPORT_STR', get_option('pgport'))
>  cdata.set_quoted('PG_KRB_SRVNAM', 'postgres')
> +if get_option('system-tzdata') != ''
> +  cdata.set_quoted('SYSTEMTZDIR', get_option('system-tzdata'))
> +endif

This doesn't quite seem sufficient - we also need to prevent building &
installing our own timezone data...

Greetings,

Andres Freund

Hi,

On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
> This currently only works on macOS.  The dtrace -G calls needed on
> other platforms are not implemented yet.

I looked into that part. The make rule passes all the backend object files as
an option, but it's not clear to me where / why that's needed. On linux it
certainly works to not pass in the object files...

Maybe CI will show problems on freebsd or such...


> Therefore, the option is not auto by default.

It probably shouldn't be auto either way, there's some overhead associated
with the probes...

Greetings,

Andres Freund

On 12.05.22 21:30, Andres Freund wrote:
> On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
>> I fixed the Perl detection issue in my macOS environment that I had reported
>> a while ago.
> 
> Hm. I wonder if it's right to check with is_file() - perhaps there are
> platforms that have split off the include directory?

The existing code uses "test -f", so using is_file() would keep it 
working the same way.

>> After that, these configure options don't have an equivalent yet:
>>
>> --disable-rpath
>> --enable-profiling
>> --disable-thread-safety
>> --with-libedit-preferred
>>
>> The first three overlap with meson built-in functionality, so we would need
>> to check whether the desired functionality is available somehow.
> 
> Which builtin functionality does --enable-profiling overlap with? There's a
> coverage option, perhaps you were thinking of that?

I saw an option about "profile guided optimization" (b_pgo), which seems 
possibly related.

> I don't think we should add --disable-thread-safety, platforms without it also
> aren't going to support ninja / meson... IIRC Thomas was planning to submit a
> patch getting rid of it independently...

sure

>>  From 049b34b6a8dd949f0eb7987cad65f6682a6ec786 Mon Sep 17 00:00:00 2001
>> From: Peter Eisentraut <peter@eisentraut.org>
>> Date: Wed, 11 May 2022 09:06:13 +0200
>> Subject: [PATCH 3/9] meson: prereq: Refactor dtrace postprocessing make rules
>>
>> Move the dtrace postprocessing sed commands into a separate file so
>> that it can be shared by meson.  Also split the rule into two for
>> proper dependency declaration.
> 
> Hm. Using sed may be problematic on windows...

This code is only used when dtrace is enabled, which probably doesn't 
apply on Windows.

On 14.05.22 01:17, Andres Freund wrote:
> On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
>> This currently only works on macOS.  The dtrace -G calls needed on
>> other platforms are not implemented yet.
> 
> I looked into that part. The make rule passes all the backend object files as
> an option, but it's not clear to me where / why that's needed. On linux it
> certainly works to not pass in the object files...
> 
> Maybe CI will show problems on freebsd or such...

Yes, it failed for me on freebsd.

>> Therefore, the option is not auto by default.
> 
> It probably shouldn't be auto either way, there's some overhead associated
> with the probes...

ok

On 2022-05-16 17:48:08 +0200, Peter Eisentraut wrote:
> On 14.05.22 01:17, Andres Freund wrote:
> > On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
> > > This currently only works on macOS.  The dtrace -G calls needed on
> > > other platforms are not implemented yet.
> > 
> > I looked into that part. The make rule passes all the backend object files as
> > an option, but it's not clear to me where / why that's needed. On linux it
> > certainly works to not pass in the object files...
> > 
> > Maybe CI will show problems on freebsd or such...
> 
> Yes, it failed for me on freebsd.

Yep, I saw those shortly after... I've implemented that bit now, although it
needs a bit more cleanup.

Hi,

On 2022-05-16 17:47:24 +0200, Peter Eisentraut wrote:
> On 12.05.22 21:30, Andres Freund wrote:
> > On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote:
> > > I fixed the Perl detection issue in my macOS environment that I had reported
> > > a while ago.
> > 
> > Hm. I wonder if it's right to check with is_file() - perhaps there are
> > platforms that have split off the include directory?
> 
> The existing code uses "test -f", so using is_file() would keep it working
> the same way.

I merged it that way. Merged.


> > >  From 049b34b6a8dd949f0eb7987cad65f6682a6ec786 Mon Sep 17 00:00:00 2001
> > > From: Peter Eisentraut <peter@eisentraut.org>
> > > Date: Wed, 11 May 2022 09:06:13 +0200
> > > Subject: [PATCH 3/9] meson: prereq: Refactor dtrace postprocessing make rules
> > > 
> > > Move the dtrace postprocessing sed commands into a separate file so
> > > that it can be shared by meson.  Also split the rule into two for
> > > proper dependency declaration.
> > 
> > Hm. Using sed may be problematic on windows...
> 
> This code is only used when dtrace is enabled, which probably doesn't apply
> on Windows.

Fair point. Merged.

Greetings,

Andres Freund

Here are some more patches that clean up various minor issues.

Hi,

On 2022-05-18 10:30:12 +0200, Peter Eisentraut wrote:
> Here are some more patches that clean up various minor issues.

I rebased the meson tree, squashed a lot of the existing commits, merged your
changes, and fixed a few more differences between autoconf and meson.


For me the difference in defines now boils down to:

- CONFIGURE_ARGS - empty in meson, not clear what to fill it with
- GETTIMEOFDAY_1ARG - test doesn't exist - I suspect it might not be necessary
- PACKAGE_STRING, PACKAGE_TARNAME - unclear if they should be implemented?
- AC_APPLE_UNIVERSAL_BUILD logic - which I don't think we need?
- pg_restrict is defined in a simplistic way
- "missing" a bunch of defines that don't appear to be referenced:
  HAVE_FSEEKO
  HAVE_GSSAPI_GSSAPI_H
  HAVE_INTTYPES_H
  HAVE_LDAP_H
  HAVE_LIBCRYPTO
  HAVE_LIBLDAP
  HAVE_LIBM
  HAVE_LIBPAM
  HAVE_LIBSSL
  HAVE_LIBXML2
  HAVE_LIBXSLT
  HAVE_MEMORY_H
  HAVE_PTHREAD
  HAVE_PTHREAD_PRIO_INHERIT
  HAVE_STDINT_H
  HAVE_STDLIB_H
  HAVE_STRING_H
  HAVE_SYS_STAT_H
  HAVE_SYS_TYPES_H
  HAVE_UNISTD_H
  SIZEOF_BOOL
  SIZEOF_OFF_T
  STDC_HEADERS
- meson additional defines, seems harmless:
  HAVE_GETTIMEOFDAY - only defined on windows rn
  HAVE_SHM_UNLINK
  HAVE_SSL_NEW
  HAVE_STRTOQ
  HAVE_STRTOUQ
  HAVE_CRYPTO_NEW_EX_DATA
- a bunch of additional #undef's


Greetings,

Andres Freund

On 18.05.22 21:48, Andres Freund wrote:
> - CONFIGURE_ARGS - empty in meson, not clear what to fill it with

Ok to leave empty for now.

> - GETTIMEOFDAY_1ARG - test doesn't exist - I suspect it might not be necessary

Might be obsolete, consider removing.

> - PACKAGE_STRING, PACKAGE_TARNAME - unclear if they should be implemented?

leave out for now

> - AC_APPLE_UNIVERSAL_BUILD logic - which I don't think we need?

no

> - "missing" a bunch of defines that don't appear to be referenced:

Yeah, looks like these are implicitly defined by some autoconf check but 
then the result is only used within configure.ac itself, so isn't needed 
afterwards.

> - meson additional defines, seems harmless:
>    HAVE_GETTIMEOFDAY - only defined on windows rn
>    HAVE_SHM_UNLINK
>    HAVE_SSL_NEW
>    HAVE_STRTOQ
>    HAVE_STRTOUQ
>    HAVE_CRYPTO_NEW_EX_DATA

Yeah, that's the opposite of the previous.

I don't see any other issues in pg_config.h either.  Obviously, some 
niche platforms might uncover some issues, but it looks good for now.

Hi Tom,

in the meson unconference session you'd spotted flex flags for psqlscanslash.l
(I think) being "hardcoded". As far as I can tell that's largely just copied
from the Makefile):

src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p

note that it's not even FLEXFLAGS += or such.

I honestly don't know enough about the various flex flags to judge what a
better approach would be? Looks like these flags are case specific? Perhaps we
could group them, i.e. have centrally defined "do compress" "don't compress"
flex flags?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> (I think) being "hardcoded". As far as I can tell that's largely just copied
> from the Makefile):

> src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p

Hmm, OK.  There *is* a FLEXFLAGS definition supplied by configure, and
I believe many of our scanners do use it, but evidently we're just
overriding it for the ones where we really care about using specific
flags.  It also looks like the configure-supplied version is usually
empty, so the fact that this variable exists may be mostly a holdover
from Autoconf practice rather than something we ever cared about.

I think the main thing I didn't like about the way you have it in the
meson file is the loss of greppability.  I could investigate this
question in a few seconds just now, but if we drop the use of
FLEXFLAGS as a macro it'll become much harder to figure out which
places use what.

            regards, tom lane

Hi,

On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> > (I think) being "hardcoded". As far as I can tell that's largely just copied
> > from the Makefile):
> 
> > src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> > src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> > src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> > src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
> 
> Hmm, OK.  There *is* a FLEXFLAGS definition supplied by configure, and
> I believe many of our scanners do use it, but evidently we're just
> overriding it for the ones where we really care about using specific
> flags.  It also looks like the configure-supplied version is usually
> empty, so the fact that this variable exists may be mostly a holdover
> from Autoconf practice rather than something we ever cared about.

Yea, it looks like that.

ISTM that it'd still be good to have something like FLEXFLAGS. But it doesn't
look great, nor really intentional, that FLEXFLAGS is overwritten rather than
appended?


> I think the main thing I didn't like about the way you have it in the
> meson file is the loss of greppability.  I could investigate this
> question in a few seconds just now, but if we drop the use of
> FLEXFLAGS as a macro it'll become much harder to figure out which
> places use what.

I disliked a bunch of repetitiveness as I had it, so I'm polishing that part
just now.

What would you want to grep for? Places that specify additional flags? Or just
places using flex?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
>> I think the main thing I didn't like about the way you have it in the
>> meson file is the loss of greppability.

> What would you want to grep for? Places that specify additional flags? Or just
> places using flex?

Well, the consistency of having a single name for "flags given to
flex" seems to me to be worth something.

            regards, tom lane

Hi Andres,

Thanks for working on this! I'm very enthusiastic about this effort and I was
glad to see on PGCon Unconference that the majority of the community seems
to be as well.

> The halfway decent list includes, I think:
> 1) cmake
> 2) bazel
> 3) meson

Was SCons considered as an option? It is widely adopted and written in Python
as well. Personally, I like the fact that with SCons you write config files
in pure Python, not some dialect you have to learn additionally. There is
a free e-book available [1].

What pros and cons do you see that make Meson a better choice?

[1]: https://scons.org/doc/production/PDF/scons-user.pdf

-- 
Best regards,
Aleksander Alekseev

Hi,

On 2022-05-26 11:47:13 +0300, Aleksander Alekseev wrote:
> Thanks for working on this! I'm very enthusiastic about this effort and I was
> glad to see on PGCon Unconference that the majority of the community seems
> to be as well.
> 
> > The halfway decent list includes, I think:
> > 1) cmake
> > 2) bazel
> > 3) meson
> 
> Was SCons considered as an option?
> What pros and cons do you see that make Meson a better choice?

I looked at it and quickly discarded it. From what I could see there's not
been meaningful moves to it in the last couple years, if anything adoption has
been dropping. And I don't think we want to end up relying on yet another half
maintained tool.

Not having a ninja backend etc didn't strike me as great either - the builds
with scons I've done weren't fast at all.

Greetings,

Andres Freund

Hi Andres,

> Not having a ninja backend etc didn't strike me as great either - the builds
> with scons I've done weren't fast at all.

I must admit, personally I never used Scons, I just know that it was considered
(an / the only?) alternative to CMake for many years. The Scons 4.3.0 release
notes say that Ninja is supported [1], but according to the user guide [2]
Ninja support is considered experimental.

Don't get me wrong, I don't insist on using Scons. I was just curious if it was
considered. Actually, a friend of mine pointed out that the fact that Scons
build files are literally a Python code could be a disadvantage. There is less
control of this code, basically it can do anything. It could complicate the
diagnosis of certain issues, etc.

Since you invested so much effort into Meson already let's just focus on it.

I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
I was going to test it against several third party extensions, but it looks like
it is a bit early for this. On Ubuntu I got the following error:

```
../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
in a function)
332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)

../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
‘PG_KEYWORD’
32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
```

On MacOS I got multiple errors regarding LDAP:

```
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/
LDAP.framework/Headers/ldap.h:1:10: error: #include nested too deeply
#include <ldap.h>

../src/interfaces/libpq/fe-connect.c:4816:2: error: use of undeclared
identifier 'LDAP'
    LDAP      *ld = NULL;
    ^

../src/interfaces/libpq/fe-connect.c:4817:2: error: use of undeclared
identifier 'LDAPMessage'
    LDAPMessage *res,
    ^
... etc...
```

I didn't invest much time into investigating these issues. For now I just
wanted to report them. Please let me know if you need any help with these
and/or additional information.

[1]: https://scons.org/scons-430-is-available.html
[2]: https://scons.org/doc/production/PDF/scons-user.pdf

--
Best regards,
Aleksander Alekseev

Hi,

On 2022-05-31 16:49:17 +0300, Aleksander Alekseev wrote:
> I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
> I was going to test it against several third party extensions, but it looks like
> it is a bit early for this. On Ubuntu I got the following error:

What do those extensions use to build? Since the unconference I added some
rudimentary PGXS compatibility, but it's definitely not complete yet.


> ```
> ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> in a function)
> 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> 
> ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> ‘PG_KEYWORD’
> 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> ```

Huh. I've not seen this before - could you provide a bit more detail about
what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
this to work.


> On MacOS I got multiple errors regarding LDAP:

Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
the rescue.  There's some crazy ordering dependency in macos framework
headers. The ldap framework contains an "ldap.h" header that includes
"ldap.h". So if you end up with the framework on the include path, you get
endless recursion.

Greetings,

Andres Freund

On 06.05.22 23:27, Andres Freund wrote:
> I added pkgconfig since then. They're not exactly the same, but pretty close,
> except for one thing: Looks like some of the ecpg libraries really should link
> to some other ecpg libs? I think we're missing something there... That then
> leads to missing requirements in the .pc files.

I took a closer look at the generated pkgconfig files.  I think they are 
ok.  There are a couple of insignificant textual differences that we 
could reduce by patching Makefile.shlib.  But technically they are ok.

There is one significant difference: the ecpg libraries now get a 
Requires.private for openssl, which I think is technically correct since 
both libpgcommon and libpgport require openssl.

Attached is a tiny patch to make the description in one file backward 
consistent.

Hi Andres,

> What do those extensions use to build? Since the unconference I added some
> rudimentary PGXS compatibility, but it's definitely not complete yet.

We mostly use CMake and Cargo, the Rust package manager. So I don't
anticipate many problems here, just want to make sure it's going to
work as expected.

> > ```
> > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > in a function)
> > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> >
> > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > ‘PG_KEYWORD’
> > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > ```
>
> Huh. I've not seen this before - could you provide a bit more detail about
> what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> this to work.

I used PIP to install Meson, since the default APT package is too old, v0.53:

$ pip3 install --user meson
$ meson --version
0.62.1
$ ninja --version
1.10.0

The branch was checked out as it was described in the first email.
Then to reproduce the issue:

$ git status
On branch meson
Your branch is up to date with 'andres/meson'.
$ git fetch andres
$ git rebase -i andres/meson
$ meson setup build --buildtype debug
$ cd build
$ ninja

This is pretty much the default Ubuntu 20.04.4 LTS system with all the
recent updates installed, so it shouldn't be a problem to reproduce
the issue with a VM.

> > On MacOS I got multiple errors regarding LDAP:
>
> Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> the rescue.  There's some crazy ordering dependency in macos framework
> headers. The ldap framework contains an "ldap.h" header that includes
> "ldap.h". So if you end up with the framework on the include path, you get
> endless recursion.

Thanks, this helped. I did the following:

$ meson configure -Dldap=disabled
$ meson configure -Dssl=openssl
$ meson configure -Dprefix=/Users/eax/pginstall
$ ninja
$ meson test
$ meson install

... and it terminated successfully. I was also able to configure and
run Postgres instance using my regular scripts, with some
modifications [1]

Then I decided to compile TimescaleDB against the newly installed
Postgres. Turns out there is a slight problem.

The extension uses CMake and also requires PostgreSQL to be compiled
with OpenSSL support. CMakeLists.txt looks for a
"--with-(ssl=)?openssl" regular expression in the "pg_config
--configure" output. The output is empty although Postgres was
compiled with OpenSSL support. The full output of pg_config looks like
this:

```
CONFIGURE =
CC = not recorded
CPPFLAGS = not recorded
CFLAGS = not recorded
CFLAGS_SL = not recorded
... etc ...
```

I get a bunch of errors from the compiler if I remove this particular
check from CMakeLists, but I have to investigate these a bit more
since the branch is based on PG15 and we don't officially support PG15
yet. It worked last time we checked a month or so ago, but the
situation may have changed.

[1]: https://github.com/afiskon/pgscripts/blob/master/single-install.sh

--
Best regards,
Aleksander Alekseev

Hi,

On 2022-06-01 06:55:06 +0200, Peter Eisentraut wrote:
> 
> On 06.05.22 23:27, Andres Freund wrote:
> > I added pkgconfig since then. They're not exactly the same, but pretty close,
> > except for one thing: Looks like some of the ecpg libraries really should link
> > to some other ecpg libs? I think we're missing something there... That then
> > leads to missing requirements in the .pc files.
> 
> I took a closer look at the generated pkgconfig files.  I think they are ok.
> There are a couple of insignificant textual differences that we could reduce
> by patching Makefile.shlib.  But technically they are ok.

Thanks for checking!


> There is one significant difference: the ecpg libraries now get a
> Requires.private for openssl, which I think is technically correct since
> both libpgcommon and libpgport require openssl.

Yea, I noticed those too. It's not great, somehow. But I don't really see a
better alternative for now.


> Attached is a tiny patch to make the description in one file backward
> consistent.

Applied.

Greetings,

Andres Freund

Hi,

On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
> 
> I used PIP to install Meson, since the default APT package is too old, v0.53:
> 
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
> 
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
> 
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
> 
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.

Will test.


> > > On MacOS I got multiple errors regarding LDAP:
> >
> > Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> > the rescue.  There's some crazy ordering dependency in macos framework
> > headers. The ldap framework contains an "ldap.h" header that includes
> > "ldap.h". So if you end up with the framework on the include path, you get
> > endless recursion.
> 
> Thanks, this helped.

Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
whether it's apple's or meson's fault.


> I did the following:
> 
> $ meson configure -Dldap=disabled
> $ meson configure -Dssl=openssl
> $ meson configure -Dprefix=/Users/eax/pginstall

FYI, you can set multiple options in one go ;)


> ... and it terminated successfully. I was also able to configure and
> run Postgres instance using my regular scripts, with some
> modifications [1]

Cool.


> Then I decided to compile TimescaleDB against the newly installed
> Postgres. Turns out there is a slight problem.
> 
> The extension uses CMake and also requires PostgreSQL to be compiled
> with OpenSSL support. CMakeLists.txt looks for a
> "--with-(ssl=)?openssl" regular expression in the "pg_config
> --configure" output. The output is empty although Postgres was
> compiled with OpenSSL support.

Makes sense. Currently we don't fill the --configure thing, because there
configure wasn't used. We could try to generate something compatible from
meson options, but I'm not sure that's a good plan.



> The full output of pg_config looks like
> this:
> 
> ```
> CONFIGURE =
> CC = not recorded
> CPPFLAGS = not recorded
> CFLAGS = not recorded
> CFLAGS_SL = not recorded
> ... etc ...
> ```
> 
> I get a bunch of errors from the compiler if I remove this particular
> check from CMakeLists, but I have to investigate these a bit more
> since the branch is based on PG15 and we don't officially support PG15
> yet. It worked last time we checked a month or so ago, but the
> situation may have changed.

I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
try to make it output something roughly compatible with the old output, for
most of those I already started to compute values for the PGXS compat stuff I
was hacking on recently.

Greetings,

Andres Freund

Hi Andres,

> Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> whether it's apple's or meson's fault.

Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
The code passes the tests too.

> > $ meson configure -Dldap=disabled
> > $ meson configure -Dssl=openssl
> > $ meson configure -Dprefix=/Users/eax/pginstall
>
> FYI, you can set multiple options in one go ;)

Thanks! ;)

> Makes sense. Currently we don't fill the --configure thing, because there
> configure wasn't used. We could try to generate something compatible from
> meson options, but I'm not sure that's a good plan.

If pg_config output was 100% backward compatible with Autotools one, that would
simplify the lives of the extension developers for sure. However, considering
that at PGCon we agreed that both Autotools and Meson will be maintained for
several releases, personally I wouldn't say that this compatibility is
necessary, nor is it realistically deliverable. Nevertheless, IMO there should
be a stable and documented way to determine the PostgreSQL version (this can be
done with `pg_config --version` for both Autotools and Meson), the build tool
used (no way to determine) and the build options (no way to determine
for Meson).

> I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> try to make it output something roughly compatible with the old output, for
> most of those I already started to compute values for the PGXS compat stuff I
> was hacking on recently.

Yes, that could explain the problem. Just for the record, I get several errors
regarding src/export.h in TimescaleDB code [1]:

```
/Users/eax/projects/c/timescaledb/src/export.h:26:5: error: pasting formed
  ')87628', an invalid preprocessing token [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
             ^
/Users/eax/projects/c/timescaledb/src/export.h:17:22: note: expanded from
  macro 'TS_EMPTY'
#define TS_EMPTY(x) (TS_CAT(x, 87628) == 87628)
                     ^
/Users/eax/projects/c/timescaledb/src/export.h:15:23: note: expanded from
  macro 'TS_CAT'
#define TS_CAT(x, y) x##y
                      ^
/Users/eax/projects/c/timescaledb/src/export.h:26:14: error: function-like
  macro '__attribute__' is not defined [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
             ^
/Users/eax/pginstall/include/postgresql/server/c.h:1339:21: note: expanded
  from macro 'PGDLLEXPORT'
#define PGDLLEXPORT __attribute__((visibility("default")))
                    ^
/Users/eax/projects/c/timescaledb/src/export.h:30:2: error: "PGDLLEXPORT is
  already defined" [clang-diagnostic-error]
#error "PGDLLEXPORT is already defined"
 ^
1 warning and 3 errors generated.
Error while processing /Users/eax/projects/c/timescaledb/src/extension.c
```

[1]: https://github.com/timescale/timescaledb/blob/main/src/export.h

-- 
Best regards,
Aleksander Alekseev

Hi,

On 2022-06-02 15:34:23 +0300, Aleksander Alekseev wrote:
> Hi Andres,
> 
> > Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> > whether it's apple's or meson's fault.
> 
> Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
> The code passes the tests too.

Cool.


> > I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> > try to make it output something roughly compatible with the old output, for
> > most of those I already started to compute values for the PGXS compat stuff I
> > was hacking on recently.
> 
> Yes, that could explain the problem. Just for the record, I get several errors
> regarding src/export.h in TimescaleDB code [1]:
> 

I think this is timescale's issue. Why are you defining / undefining
PGDLLEXPORT?

Part of the patch series is to use visibility attributes, and your #if
TS_EMPTY(PGDLLEXPORT) thing can't handle that.

Greetings,

Andres Freund

Hi,

On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
> 
> I used PIP to install Meson, since the default APT package is too old, v0.53:
> 
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
> 
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
> 
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
> 
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.

Chatting with a colleague (who unbeknownst to me hit something similar in the
past) I think we figured it out. It's not due to Ubuntu 20.04 or such. It's
likely due to previously having an in-tree build with autoconf, doing make
clean, doing a git pull, then building with meson. The meson build doesn't yet
handle pre-existing flex / bison output.

I had tried to defend against conflicts with in-tree builds by detecting an
in-tree pg_config.h, but that doesn't help with files that aren't removed by
make clean. Like bison / flex output.

And I didn't notice this problem because it doesn't cause visible issues until
the lexer / grammar changes...


I'm not quite sure what the proper behaviour is when doing an out-of-tree
build with meson (all builds are out-of-tree), with a pre-existing flex /
bison output in the source tree that is out of date.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I'm not quite sure what the proper behaviour is when doing an out-of-tree
> build with meson (all builds are out-of-tree), with a pre-existing flex /
> bison output in the source tree that is out of date.

Definitely sounds like a gotcha.

On the one hand, there's been some discussion already of removing all
derived files from tarballs and just insisting that users provide all
needed tools when building from source.  If we did that, it could be
sufficient for the meson build to check that no such files are present
in the source tree.  (Checking a couple of them would be enough, likely.)

On the other hand, I'm not sure that I want such a change to be forced
by a toolchain change.  It definitely seems a bit contrary to the plan
we'd formed of allowing meson and make-based builds to coexist for
a few years, because we'd be breaking at least some make-based build
processes.

Could we have the meson build check that, say, if gram.c exists it
is newer than gram.y?  Or get it to ignore an in-tree gram.c?

            regards, tom lane

Hi,

On 2022-06-02 13:08:49 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I'm not quite sure what the proper behaviour is when doing an out-of-tree
> > build with meson (all builds are out-of-tree), with a pre-existing flex /
> > bison output in the source tree that is out of date.
> 
> Definitely sounds like a gotcha.
> 
> On the one hand, there's been some discussion already of removing all
> derived files from tarballs and just insisting that users provide all
> needed tools when building from source.  If we did that, it could be
> sufficient for the meson build to check that no such files are present
> in the source tree.  (Checking a couple of them would be enough, likely.)

There already is a check for pg_config.h, so the most obvious source of this
is addressed. Just didn't think about the files that make clean doesn't remove
:/.


> On the other hand, I'm not sure that I want such a change to be forced
> by a toolchain change.  It definitely seems a bit contrary to the plan
> we'd formed of allowing meson and make-based builds to coexist for
> a few years, because we'd be breaking at least some make-based build
> processes.

Agreed. I think it'd be pretty reasonable to not include flex / bison
output. They're not hard to acquire. The docs are perhaps another story.

I think it might be fine to say that make reallyclean (*) is required if
there's some conflicting in-source tree file?


> Could we have the meson build check that, say, if gram.c exists it
> is newer than gram.y?  Or get it to ignore an in-tree gram.c?

I suspect the problem with ignoring is gram.h, that's probably a bit harder to
ignore. Right now I'm leaning towards either always erroring out if there's
bison/flex output in the source tree (with a hint towards make
reallyclean(*)), or erroring out if they're out of date (again with a hint
towards reallyclean)?

Alternatively we could just remove the generated .c/h files from the source
dir, as a part of regenerating them in the build dir? But I like the idea of
the source dir being readonly outside of explicit targets modifying sources
(e.g. update-unicode or such).

Greetings,

Andres Freund

(*) do we really not have a target that removes bison / flex output?

Andres Freund <andres@anarazel.de> writes:
> (*) do we really not have a target that removes bison / flex output?

maintainer-clean

            regards, tom lane

Hi,

On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > (*) do we really not have a target that removes bison / flex output?
> 
> maintainer-clean

Don't think so:

# gram.c, gram.h, and scan.c are in the distribution tarball, so they
# are not cleaned here.
clean distclean maintainer-clean:
    rm -f lex.backup

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> (*) do we really not have a target that removes bison / flex output?

>> maintainer-clean

> Don't think so:

See about line 300 in src/backend/Makefile.  In any case, it's
easy to show by experiment that it does.

$ make maintainer-clean
...
$ git status --ignored
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

            regards, tom lane

Hi,

On 2022-06-02 15:05:10 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> >> Andres Freund <andres@anarazel.de> writes:
> >>> (*) do we really not have a target that removes bison / flex output?
> 
> >> maintainer-clean
> 
> > Don't think so:
> 
> See about line 300 in src/backend/Makefile.  In any case, it's
> easy to show by experiment that it does.
> 
> $ make maintainer-clean
> ...
> $ git status --ignored
> On branch master
> Your branch is up to date with 'origin/master'.
> 
> nothing to commit, working tree clean

Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
see it getting cleaned up.

It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
clean them up.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> see it getting cleaned up.

> It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> clean them up.

I agree the factorization of this ain't great.  I'd think about improving
it, were it not that we're trying to get rid of it.

(But with meson, the whole idea of building or cleaning just part of the
tree is out the window anyway, no?)

            regards, tom lane

Hi,

On 2022-06-02 15:53:50 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> > see it getting cleaned up.
> 
> > It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> > that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> > clean them up.
> 
> I agree the factorization of this ain't great.  I'd think about improving
> it, were it not that we're trying to get rid of it.

+1. I think I just wanted to excuse my confusion...


> (But with meson, the whole idea of building or cleaning just part of the
> tree is out the window anyway, no?)

Cleaning parts of the tree isn't supported as far as I know (not that I've
needed it). You can build parts of the tree by specifying the target
(e.g. ninja src/backend/postgres) or by specifying meta-targets (e.g. ninja
contrib backend). I've thought about contributing a patch to meson to
automatically generate targets for each directory that has sub-targets - it's
just a few lines.

Greetings,

Andres Freund

Hi hackers,

> See about line 300 in src/backend/Makefile. In any case, it's
> easy to show by experiment that it does.

`make maintainer-clean` did the trick, thanks. I suggest modifying meson.build
accordingly:

-run make distclean in the source tree.
+run `make maintainer-clean` in the source tree.

> I think this is timescale's issue. Why are you defining / undefining
> PGDLLEXPORT?

That's a great question.

As I understand some time ago the developers had a problem with a collision of
exported symbols on *nix platforms [1] and chose to solve it by re-defining
PGDLLEXPORT to __attribute__((visibility ("default"))) for GCC and CLang.
I agree that this is a questionable approach. Redefining a macro provided
by Postgres doesn't strike me as a good idea. I tried to remove this
re-definition, but it didn't go well [2]. So apparently it should be addressed
somehow differently.

> Part of the patch series is to use visibility attributes, and your #if
> TS_EMPTY(PGDLLEXPORT) thing can't handle that.

Out of curiosity, how come a patchset that adds an alternative build system
changes the visibility attributes? I would guess they should be the same
for both Autotools and Meson. Is it necessary in order to make Meson work?
If not, maybe it should be a separate patch.

[1]: https://github.com/timescale/timescaledb/commit/027b7b29420a742d7615c70d9f19b2b99c488c2c
[2]: https://github.com/timescale/timescaledb/pull/4413

-- 
Best regards,
Aleksander Alekseev

Hi,

On 2022-06-03 12:35:45 +0300, Aleksander Alekseev wrote:
> > Part of the patch series is to use visibility attributes, and your #if
> > TS_EMPTY(PGDLLEXPORT) thing can't handle that.
> 
> Out of curiosity, how come a patchset that adds an alternative build system
> changes the visibility attributes?

It was the simplest path - on windows (and AIx) extension symbols need to be
explicitly exported. We did that by building the objects constituting
extension libraries, collecting the symbols, generating an export file with
all the symbols, which then is passed to the linker.  It was a lot less work
to just add the necessary PGDLLEXPORT annotations than make that export file
generation work for extensions.


> I would guess they should be the same for both Autotools and Meson.

It is, the patch adds it to both.


> Is it necessary in order to make Meson work?

Yes, or at least the simplest path.


> If not, maybe it should be a separate patch.

It is.

https://commitfest.postgresql.org/38/3396/

Greetings,

Andres Freund

On Tue, May 24, 2022 at 08:08:26PM +0200, Peter Eisentraut wrote:
> On 18.05.22 21:48, Andres Freund wrote:
>> - GETTIMEOFDAY_1ARG - test doesn't exist - I suspect it might not be necessary
>
> Might be obsolete, consider removing.

I just came across this one independently of what you are doing for
meson, and based on a lookup of the buildfarm, I think that it can be
removed.  One reference about GETTIMEOFDAY_1ARG on the -hackers list
comes from here, from 20 years ago:
https://www.postgresql.org/message-id/a1eeu5$1koe$1@news.tht.net
--
Michael

I looked at some of the "prereq" patches again to see what state they 
are in:

commit 351a12f48e395b31cce4aca239b934174b36ea9d
Author: Andres Freund <andres@anarazel.de>
Date:   Wed Apr 20 22:46:54 2022

     prereq: deal with \ paths in basebackup_to_shell tests.

This is a new component in PG15, so a fix might be in scope for PG15 
too.  But I don't know if this change is really necessary.  There are 
other tests that use the GZIP and TAR environment variables (e.g., 
pg_verifybackup).  If this is a problem there too, we should think of a 
general solution.  If not, it could use some explanation.


commit c00642483a53f4ee6e351085c7628363c293ee61
Author: Andres Freund <andres@anarazel.de>
Date:   Fri Mar 25 21:44:48 2022

     meson: prereq: unicode: allow to specify output directory.

OK with attached fixup (but see below).


commit 31313056e153e099f236a29b752f7610c4f7764f
Author: Andres Freund <andres@anarazel.de>
Date:   Thu Jan 20 08:36:50 2022

     meson: prereq: generate-errcodes.pl: accept output file

This is ok, but seems unnecessary, since meson can capture the output of 
a single file.  (See also similar script generate-errcodes-table.pl in 
doc/, which uses capture.)


commit e4e77c0e20f3532be4ed270a7cf8b965b7cafa49
Author: Andres Freund <andres@anarazel.de>
Date:   Thu Jan 20 08:36:50 2022

     meson: prereq: add output path arg in generate-lwlocknames.pl

We should make the command-line interface here the same as the unicode 
script: Either make the output directory a positional argument or an 
option. I don't have a strong feeling about it either way, but perhaps 
the solution with the option is more elegant and would also not require 
changing the makefiles.  Also, we should decide on short or long option: 
The code declares a long option, but the build uses a short option. 
It's confusing that that even works.


commit 7866620afa65223f6e657da972f501615fd32d3b
Author: Andres Freund <andres@anarazel.de>
Date:   Wed Apr 20 21:01:31 2022

     meson: prereq: output and depencency tracking work.

This could be split into multiple parts with more detailed explanations. 
  I see where you're going but not everything is fully clear to me 
(especially the guc-file.c.h stuff).

Attached is a patch the finishes up the work to move the snowball SQL 
script generation into a separate script.

Hi,

On 2022-06-08 08:27:06 +0200, Peter Eisentraut wrote:
> I looked at some of the "prereq" patches again to see what state they are
> in:
> 
> commit 351a12f48e395b31cce4aca239b934174b36ea9d
> Author: Andres Freund <andres@anarazel.de>
> Date:   Wed Apr 20 22:46:54 2022
> 
>     prereq: deal with \ paths in basebackup_to_shell tests.
> 
> This is a new component in PG15, so a fix might be in scope for PG15 too.

Yea, I should probably post that to the relevant thread. I think at that point
I was just trying to get a rebase not to fail anymore...


> But I don't know if this change is really necessary.  There are other tests
> that use the GZIP and TAR environment variables (e.g., pg_verifybackup).  If
> this is a problem there too, we should think of a general solution.  If not,
> it could use some explanation.

I got failures on windows without it - which we just don't see on windows
because currently nothing runs these tests :(. The pg_verifybackup case likely
is unproblematic because it uses the array form of building subcommands,
instead of string interpolation.


> commit c00642483a53f4ee6e351085c7628363c293ee61
> Author: Andres Freund <andres@anarazel.de>
> Date:   Fri Mar 25 21:44:48 2022
> 
>     meson: prereq: unicode: allow to specify output directory.
> 
> OK with attached fixup (but see below).

Merged.


> commit 31313056e153e099f236a29b752f7610c4f7764f
> Author: Andres Freund <andres@anarazel.de>
> Date:   Thu Jan 20 08:36:50 2022
> 
>     meson: prereq: generate-errcodes.pl: accept output file
> 
> This is ok, but seems unnecessary, since meson can capture the output of a
> single file.  (See also similar script generate-errcodes-table.pl in doc/,
> which uses capture.)

Not sure why I didn't do that. It might be because the meson capture stuff has
a noticable overhead, particularly on windows, because it starts up a python
interpreter. Since nearly the whole build depends on generate-errcodes.pl to
have run...


> commit e4e77c0e20f3532be4ed270a7cf8b965b7cafa49
> Author: Andres Freund <andres@anarazel.de>
> Date:   Thu Jan 20 08:36:50 2022
> 
>     meson: prereq: add output path arg in generate-lwlocknames.pl
> 
> We should make the command-line interface here the same as the unicode
> script: Either make the output directory a positional argument or an option.
> I don't have a strong feeling about it either way, but perhaps the solution
> with the option is more elegant and would also not require changing the
> makefiles.

I don't really have an opinion what's better here, so I'll go with your
preference / the option.


> Also, we should decide on short or long option: The code
> declares a long option, but the build uses a short option. It's confusing
> that that even works.

Getopt::Long auto-generates short options afaict...


> commit 7866620afa65223f6e657da972f501615fd32d3b
> Author: Andres Freund <andres@anarazel.de>
> Date:   Wed Apr 20 21:01:31 2022
> 
>     meson: prereq: output and depencency tracking work.
> 
> This could be split into multiple parts with more detailed explanations.  I
> see where you're going but not everything is fully clear to me (especially
> the guc-file.c.h stuff).

Will take a stab at doing so.


> From 51c6d3544ae9e652c7aac26102a8bf5a116fb182 Mon Sep 17 00:00:00 2001
> From: Peter Eisentraut <peter@eisentraut.org>
> Date: Tue, 7 Jun 2022 22:54:26 +0200
> Subject: [PATCH] fixup! meson: prereq: unicode: allow to specify output
>  directory.

Merged.

Greetings,

Andres Freund

Hi,

On 2022-06-08 14:33:16 +0200, Peter Eisentraut wrote:
> Attached is a patch the finishes up the work to move the snowball SQL script
> generation into a separate script.

That looks good, merged. I did split the commit, because there's not yet a
meson.build "at the time" of the prereq: commits.

One thing I'm not quite sure about: Why does the makefile need awareness of
the stop files, but Install.pm doesn't? I suspect currently the patch leads to
stopwords not being installed on windows anymore?

Greetings,

Andres Freund

On 14.06.22 20:27, Andres Freund wrote:
> One thing I'm not quite sure about: Why does the makefile need awareness of
> the stop files, but Install.pm doesn't? I suspect currently the patch leads to
> stopwords not being installed on windows anymore?

Install.pm contains this elsewhere:

         GenerateTsearchFiles($target);
         CopySetOfFiles(
             'Stopword files',
             [ glob("src\\backend\\snowball\\stopwords\\*.stop") ],
             $target . '/share/tsearch_data/');

It's a bit confusing that the "generate" function that we are patching 
also installs some of the files right away, while the rest is installed 
by the calling function.

On 2022-06-14 20:47:59 +0200, Peter Eisentraut wrote:
> On 14.06.22 20:27, Andres Freund wrote:
> > One thing I'm not quite sure about: Why does the makefile need awareness of
> > the stop files, but Install.pm doesn't? I suspect currently the patch leads to
> > stopwords not being installed on windows anymore?
> 
> Install.pm contains this elsewhere:
> 
>         GenerateTsearchFiles($target);
>         CopySetOfFiles(
>             'Stopword files',
>             [ glob("src\\backend\\snowball\\stopwords\\*.stop") ],
>             $target . '/share/tsearch_data/');
> 
> It's a bit confusing that the "generate" function that we are patching also
> installs some of the files right away, while the rest is installed by the
> calling function.

Ugh, that's confusing indeed. Thanks for the explanation.

Hi,

Attached is an updated version of the meson patchset. There has been a steady
stream of incremental work over the last month, with patches from Peter
Eisentraut and Nazir Yavuz.

I tried to address the review comments Peter had downthread about the prep
patches. The one that I know is still outstanding is that there's still
different ways of passing output directories as parameters to a bunch of
scripts added, will resolve that next (some have been fixed).

Now the patchset contains a, somewhat hacky and incomplete, implementation of
pgxs, even when using meson. Basically a compatible Makefile.global.in is
generated.

There's a lot of small and medium sized changes.

As usual the changes are also my git branch [1], which gets updated fairly
regularly.

Greetings,

Andres Freund

[1] https://github.com/anarazel/postgres/tree/meson

vcvarsall isn't needed in cirrus' "check_world" scripts.

I'm missing any way to re/run cirrus only for msbuild OR ninja OR homegrown
with something more granular than "ci-os-only: windows".  (The same thing
applies to the mingw patch).

I'll mail shortly about ccache.

-- 
Justin

Hi,

On 2022-07-01 14:01:11 -0500, Justin Pryzby wrote:
> vcvarsall isn't needed in cirrus' "check_world" scripts.

E.g. for the ecpg tests it isn't, except that we don't currently build ecpg on
windows. But I plan to fix that.


> I'm missing any way to re/run cirrus only for msbuild OR ninja OR homegrown
> with something more granular than "ci-os-only: windows".  (The same thing
> applies to the mingw patch).

Not sure that's really worth adding - I don't forsee merging all those
tasks. But I'm open to proposals.


> I'll mail shortly about ccache.

There's a meson PR that should fix some of the issues, need to test it...

Greetings,

Andres Freund

On 01.07.22 11:33, Andres Freund wrote:
> Attached is an updated version of the meson patchset. There has been a steady
> stream of incremental work over the last month, with patches from Peter
> Eisentraut and Nazir Yavuz.
> 
> I tried to address the review comments Peter had downthread about the prep
> patches. The one that I know is still outstanding is that there's still
> different ways of passing output directories as parameters to a bunch of
> scripts added, will resolve that next (some have been fixed).

Here is my rough assessment of where we are with this patch set:

08b4330ded prereq: deal with \ paths in basebackup_to_shell tests.

This still needs clarification, per my previous review.

3bf5b317d5 meson: prereq: Specify output directory for psql sql help script.
2e5ed807f8 meson: prereq: ecpg: add and use output directory argument 
for parse.pl.
4e7fab01c5 meson: prereq: msvc: explicit output file for pgflex.pl
cdcd3da4c4 meson: prereq: add output path arg in generate-lwlocknames.pl
1f655486e4 meson: prereq: generate-errcodes.pl: accept output file
e834c48758 meson: prereq: unicode: allow to specify output directory.

You said you are still finalizing these.  I think we can move ahead with 
these once that is done.

9f4a9b1749 meson: prereq: move snowball_create.sql creation into perl file.

This looks ready, except I think it needs to be hooked into the
distprep target, since it's now a Perl script running at build time.

8951a6721e meson: prereq: Refactor dtrace postprocessing make rules

This looks ready.

bda6a45bae meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build

I understand the intention behind this, but I think it changes the
behavior in an undesirable way.  Before this patch, you can go into
src/test/ssl/ and run make check manually.  This was indeed the only
way to do it before PG_TEST_EXTRA.  With this patch, this would now
skip all the tests unless you set PG_TEST_EXTRA, even if you run the
specific test directly.

I think this needs a different idea.

eb852cc023 meson: prereq: Can we get away with not export-all'ing libraries?

This is also at <https://commitfest.postgresql.org/38/3396/>, which
hasn't seen any activity in a while.  I think this needs a resolution
one way or the other before we can proceed to the main act.

2cc276ced6 meson: prereq: add src/tools/gen_versioning_script.pl.

Note that in the make build system we can only use perl before
distprep.  So it's not clear whether a script like this would help
unify the code.  Of course, we could still use it with the
understanding that it will be separate.

351ac51a89 meson: prereq: remove LLVM_CONFIG from Makefile.global.in

This can be committed.  AFAICT, LLVM_CONFIG is only used within
configure.

dff7b5a960 meson: prereq: regress: allow to specify director containing 
expected files.

This could use a bit more explanation, but it doesn't look
controversial so far.

243f99da38 wip: split TESTDIR into two.

This one has already caused a bit of confusion, but the explanation at


https://www.postgresql.org/message-id/flat/20220601211112.td2ato4wjqf7afnv%40alap3.anarazel.de#1f250dee73cf0da29a6d2c020c3bde08

seems reasonable.  But it clearly needs further work.

88dd280835 meson: Add meson based buildsystem.
1ee3073a3c meson: ci: Build both with meson and as before.

These are for later. ;-)

In the meantime, also of interest to this effort:

- If we're planning to remove the postmaster symlink in PG16, maybe we
   should start a discussion on that.

- This patch is for unifying the list of languages in NLS, as
   previously discussed: https://commitfest.postgresql.org/38/3737/

Hi

On 2022-07-06 11:03:31 +0200, Peter Eisentraut wrote:
> On 01.07.22 11:33, Andres Freund wrote:
> > Attached is an updated version of the meson patchset. There has been a steady
> > stream of incremental work over the last month, with patches from Peter
> > Eisentraut and Nazir Yavuz.
> > 
> > I tried to address the review comments Peter had downthread about the prep
> > patches. The one that I know is still outstanding is that there's still
> > different ways of passing output directories as parameters to a bunch of
> > scripts added, will resolve that next (some have been fixed).
> 
> Here is my rough assessment of where we are with this patch set:
> 
> 08b4330ded prereq: deal with \ paths in basebackup_to_shell tests.
> 
> This still needs clarification, per my previous review.

Hm. I thought I had explained that bit, but apparently not. Well, it's pretty
simple - without this, the test fail on windows for me, as soon as one of the
binaries is in a directory with spaces (which is common on windows). Iimagine
what happens with e.g.
  qq{$gzip --fast > "$escaped_backup_path\\\\%f.gz"}
if $gzip contains spaces.


This doesn't happen currently on CI because nothing runs these tests on
windows yet.


> bda6a45bae meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build
> 
> I understand the intention behind this, but I think it changes the
> behavior in an undesirable way.  Before this patch, you can go into
> src/test/ssl/ and run make check manually.  This was indeed the only
> way to do it before PG_TEST_EXTRA.  With this patch, this would now
> skip all the tests unless you set PG_TEST_EXTRA, even if you run the
> specific test directly.

It's not a free lunch, I agree. But I think the downsides outweigh the upsides
by far. Not seeing that tests were skipped in the test output is quite
problematic imo. And with meson's testrunner we're going to need something
that deals with skipping these tests - and it's more important to have them
skipped, so that they show up in the test summary.

It's not like it's hard to set PG_TEST_EXTRA for a single command invocation?



> 243f99da38 wip: split TESTDIR into two.
> 
> This one has already caused a bit of confusion, but the explanation at
> 
>
https://www.postgresql.org/message-id/flat/20220601211112.td2ato4wjqf7afnv%40alap3.anarazel.de#1f250dee73cf0da29a6d2c020c3bde08
> 
> seems reasonable.  But it clearly needs further work.

Yea. I kind of want to get some of the preparatory stuff out of the way first.


> 88dd280835 meson: Add meson based buildsystem.
> 1ee3073a3c meson: ci: Build both with meson and as before.
> 
> These are for later. ;-)
> 
> In the meantime, also of interest to this effort:
> 
> - If we're planning to remove the postmaster symlink in PG16, maybe we
>   should start a discussion on that.

Yea.


> - This patch is for unifying the list of languages in NLS, as
>   previously discussed: https://commitfest.postgresql.org/38/3737/

There seems little downside to doing so, so ...

Greetings,

Andres Freund

On 06.07.22 15:21, Andres Freund wrote:
>> Here is my rough assessment of where we are with this patch set:
>>
>> 08b4330ded prereq: deal with \ paths in basebackup_to_shell tests.
>>
>> This still needs clarification, per my previous review.
> Hm. I thought I had explained that bit, but apparently not. Well, it's pretty
> simple - without this, the test fail on windows for me, as soon as one of the
> binaries is in a directory with spaces (which is common on windows). Iimagine
> what happens with e.g.
>    qq{$gzip --fast > "$escaped_backup_path\\\\%f.gz"}
> if $gzip contains spaces.
> 
> 
> This doesn't happen currently on CI because nothing runs these tests on
> windows yet.

Hmm, maybe this patch looked different the last time I saw it.  I see 
your point.

The quoting of "$gzip" is clearly necessary.

What about the backslash replacements s{\\}{/}g ?  Is that also required 
for passing the path through the shell?  If so, the treatment of $tar in 
that way doesn't seem necessary, since that doesn't get called through 
an intermediate shell.  (That would then also explain why $gzip in the 
pg_basebackup tests doesn't require that treatment, which had previously 
confused me.)

If my understanding of this is correct, then I suggest:

1. Add a comment to the $gzip =~ s{\\}{/}g ... line.
2. Remove the $tar =~ s{\\}{/}g ... line.
3. Backpatch to PG15.

On 06.07.22 15:21, Andres Freund wrote:
>> - This patch is for unifying the list of languages in NLS, as
>>    previously discussed:https://commitfest.postgresql.org/38/3737/
> There seems little downside to doing so, so ...

This has been committed, so on the next rebase, the languages arguments 
can be removed from the i18n.gettext() calls.

On 06.07.22 15:21, Andres Freund wrote:
>> bda6a45bae meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build
>>
>> I understand the intention behind this, but I think it changes the
>> behavior in an undesirable way.  Before this patch, you can go into
>> src/test/ssl/ and run make check manually.  This was indeed the only
>> way to do it before PG_TEST_EXTRA.  With this patch, this would now
>> skip all the tests unless you set PG_TEST_EXTRA, even if you run the
>> specific test directly.
> It's not a free lunch, I agree. But I think the downsides outweigh the upsides
> by far. Not seeing that tests were skipped in the test output is quite
> problematic imo. And with meson's testrunner we're going to need something
> that deals with skipping these tests - and it's more important to have them
> skipped, so that they show up in the test summary.
> 
> It's not like it's hard to set PG_TEST_EXTRA for a single command invocation?

It's probably ok.  I have it set in my environment all the time anyway, 
so it wouldn't affect me.  But it's the sort of thing people might be 
particular about, so I thought it was worth pointing out.

Hi,

On 2022-07-07 12:09:32 +0200, Peter Eisentraut wrote:
> On 06.07.22 15:21, Andres Freund wrote:
> > > Here is my rough assessment of where we are with this patch set:
> > > 
> > > 08b4330ded prereq: deal with \ paths in basebackup_to_shell tests.
> > > 
> > > This still needs clarification, per my previous review.
> > Hm. I thought I had explained that bit, but apparently not. Well, it's pretty
> > simple - without this, the test fail on windows for me, as soon as one of the
> > binaries is in a directory with spaces (which is common on windows). Iimagine
> > what happens with e.g.
> >    qq{$gzip --fast > "$escaped_backup_path\\\\%f.gz"}
> > if $gzip contains spaces.
> > 
> > 
> > This doesn't happen currently on CI because nothing runs these tests on
> > windows yet.
> 
> Hmm, maybe this patch looked different the last time I saw it.

Don't think it had changed since I wrote it first.


> The quoting of "$gzip" is clearly necessary.
> 
> What about the backslash replacements s{\\}{/}g ?  Is that also required for
> passing the path through the shell?

I don't recall the details, but it's definitely needed for embedding it into
postgresql.conf. We could double the escapes instead, but that doesn't seem an
improvement (and wouldn't work when passing it to the shell anymore).


> If so, the treatment of $tar in that
> way doesn't seem necessary, since that doesn't get called through an
> intermediate shell.  (That would then also explain why $gzip in the
> pg_basebackup tests doesn't require that treatment, which had previously
> confused me.)

Yea, it doesn't immediately look like it's needed, and the test passes without
it. I guess I might just have tried to be complete...

Greetings,

Andres Freund

Hi,

Attached is v10 of the meson patchset. Lots of small changes, I don't think
anything major. I tried to address most of Peter's feedback for the earlier
patches.

After this I plan to clean up the "export" patch, since that's I think the
next bigger step, and an improvement on its own. The step after will be to
discuss where we want the output of tests to reside, whether the naming scheme
for tests is good etc.

I did try to address Peter's criticism around inconsistency of the added
parameters to perl scripts. I hope it's more consistent now. I used the
opportunity to make src/tools/msvc use the "output directory" parameters,
providing coverage for those paths (and removing a few unnecessary chdirs, but
...).

Greetings,

Andres Freund

Hi,

On 2022-07-13 08:39:45 +0200, Peter Eisentraut wrote:
> On 06.07.22 15:21, Andres Freund wrote:
> > > - This patch is for unifying the list of languages in NLS, as
> > >    previously discussed:https://commitfest.postgresql.org/38/3737/
> > There seems little downside to doing so, so ...
> 
> This has been committed, so on the next rebase, the languages arguments can
> be removed from the i18n.gettext() calls.

That's done in v10 I posted yesterday.


On 2022-07-06 11:03:31 +0200, Peter Eisentraut wrote:
> 3bf5b317d5 meson: prereq: Specify output directory for psql sql help script.
> 2e5ed807f8 meson: prereq: ecpg: add and use output directory argument for
> parse.pl.
> 4e7fab01c5 meson: prereq: msvc: explicit output file for pgflex.pl
> cdcd3da4c4 meson: prereq: add output path arg in generate-lwlocknames.pl
> 1f655486e4 meson: prereq: generate-errcodes.pl: accept output file
> e834c48758 meson: prereq: unicode: allow to specify output directory.
> 
> You said you are still finalizing these.  I think we can move ahead with
> these once that is done.

I tried to address the pending things in these patches.


> 9f4a9b1749 meson: prereq: move snowball_create.sql creation into perl file.
> 
> This looks ready, except I think it needs to be hooked into the
> distprep target, since it's now a Perl script running at build time.

Done. I vacillated between doing the distprep rule in
src/backend/snowball/Makefile (as most things outside of the backend do) and
src/backend/Makefile (most backend things). Ended up with doing it in
snowball/Makefile.

Greetings,

Andres Freund

Hi Andres,

> Attached is v10 of the meson patchset. Lots of small changes, I don't think
> anything major. I tried to address most of Peter's feedback for the earlier
> patches.
>
> After this I plan to clean up the "export" patch, since that's I think the
> next bigger step, and an improvement on its own. The step after will be to
> discuss where we want the output of tests to reside, whether the naming scheme
> for tests is good etc.
>
> I did try to address Peter's criticism around inconsistency of the added
> parameters to perl scripts. I hope it's more consistent now. I used the
> opportunity to make src/tools/msvc use the "output directory" parameters,
> providing coverage for those paths (and removing a few unnecessary chdirs, but
> ...).

Thanks for continuing to work on this!

Just a quick question - is there a reason for changing the subject of
the emails?

Not all email clients handle this well, e.g. Google Mail considers
this being 10 separate threads. The CF application and/or
pgsql-hackers@ archive also don't recognise this as a continuation of
the original thread. So all the discussions in -v8, -v9, -v9 ets
threads get lost.

May I suggest using a single thread?

-- 
Best regards,
Aleksander Alekseev

Hi again,

> Just a quick question - is there a reason for changing the subject of
> the emails?
>
> Not all email clients handle this well, e.g. Google Mail considers
> this being 10 separate threads. The CF application and/or
> pgsql-hackers@ archive also don't recognise this as a continuation of
> the original thread. So all the discussions in -v8, -v9, -v9 ets
> threads get lost.
>
> May I suggest using a single thread?

OK, the part about the archive is wrong - I scrolled right to the end
of the thread, didn't notice v10 patch above and assumed it was lost.
Sorry for the confusion. However, the part about various email clients
is accurate.

-- 
Best regards,
Aleksander Alekseev

On 15.07.22 07:08, Andres Freund wrote:
> Attached is v10 of the meson patchset. Lots of small changes, I don't think
> anything major. I tried to address most of Peter's feedback for the earlier
> patches.

The following patches are ok to commit IMO:

a1c5542929 prereq: Deal with paths containing \ and spaces in basebackup_to_shell tests
e37951875d meson: prereq: psql: Output dir and dependency generation for sql_help
18cc9fbd02 meson: prereq: ecpg: Add and use output directory argument for preproc/*.pl
58a32694e9 meson: prereq: Move snowball_create.sql creation into perl file
59b8bffdaf meson: prereq: Add output path arg in generate-lwlocknames.pl
2db97b00d5 meson: prereq: generate-errcodes.pl: Accept output file
fb8f52f21d meson: prereq: unicode: Allow to specify output directory
8f1e4410d6 meson: prereq: Refactor dtrace postprocessing make rules
3d18a20b11 meson: prereq: Add --outdir option to gen_node_support.pl

Hi,

On 2022-07-18 11:12:10 +0300, Aleksander Alekseev wrote:
> > Just a quick question - is there a reason for changing the subject of
> > the emails?
> >
> > Not all email clients handle this well, e.g. Google Mail considers
> > this being 10 separate threads. The CF application and/or
> > pgsql-hackers@ archive also don't recognise this as a continuation of
> > the original thread. So all the discussions in -v8, -v9, -v9 ets
> > threads get lost.
> >
> > May I suggest using a single thread?
> 
> OK, the part about the archive is wrong - I scrolled right to the end
> of the thread, didn't notice v10 patch above and assumed it was lost.
> Sorry for the confusion. However, the part about various email clients
> is accurate.

For me the thread is too long to look through without some separation. I
wouldn't do the version in the subject for a small patchset / thread, but at
this size I think it's reasonable.

Greetings,

Andres Freund

Hi,

On 2022-07-18 11:33:09 +0200, Peter Eisentraut wrote:
> The following patches are ok to commit IMO:
> 
> a1c5542929 prereq: Deal with paths containing \ and spaces in basebackup_to_shell tests
> e37951875d meson: prereq: psql: Output dir and dependency generation for sql_help
> 18cc9fbd02 meson: prereq: ecpg: Add and use output directory argument for preproc/*.pl
> 58a32694e9 meson: prereq: Move snowball_create.sql creation into perl file
> 59b8bffdaf meson: prereq: Add output path arg in generate-lwlocknames.pl
> 2db97b00d5 meson: prereq: generate-errcodes.pl: Accept output file
> fb8f52f21d meson: prereq: unicode: Allow to specify output directory
> 8f1e4410d6 meson: prereq: Refactor dtrace postprocessing make rules
> 3d18a20b11 meson: prereq: Add --outdir option to gen_node_support.pl

I pushed these. Thanks for the reviews and patches!

The symbol export stuff has also been pushed (discussed in a separate thread).

It's nice to see the meson patchset length reduced by this much.

I pushed a rebased version of the remaining branches to git. I'll be on
vacation for a bit, I'm not sure I can get a new version with further cleanups
out before.


Given that we can't use src/tools/gen_versioning_script.pl for the make build,
due to not depending on perl for tarball builds, I'm inclined to rewrite it
python (which we depend on via meson anyway) and consider it a meson specific
wrapper?


Bilal, Peter previously commented on the pg_regress change for ecpg, perhaps
you can comment on that?

In https://postgr.es/m/0e81e45c-c9a5-e95b-2782-ab2dfec8bf57%40enterprisedb.com
On 2022-07-06 11:03:31 +0200, Peter Eisentraut wrote:
> dff7b5a960 meson: prereq: regress: allow to specify director containing
> expected files.
> 
> This could use a bit more explanation, but it doesn't look
> controversial so far.

Greetings,

Andres Freund

Hi,

On 2021-10-31 16:24:48 -0700, Andres Freund wrote:
> - support for building docs.
>   I couldn't get dbtoepub work in a vpath style build, so I changed that
>   to also use pandoc. No idea if anybody uses the epub rules?

combing through various FIXMEs in the meson patch I started to look into docs
again. A few questions / noteworthy points:

- I still haven't gotten dbtoepub to work in vpath style builds (epub
  generation is instead using pandoc). Could somebody using these comment on
  the quality difference?

  We don't seem to offer these for download anywhere...

  Worth noting that dbtoepub takes approximately forever (>25min). Not that
  pandoc is fast, but ...

  Unfortunately pandoc doesn't understand <part> level stuff. That'd probably
  easy enough to tweak, but ...

- We have logic to change the man section (grep for sqlmansect) for some
  platforms. The only remaining platform is solaris. I'm inclined to not
  implement that.

- I've not implemented the texinfo targets - don't think they're really used?

Greetings,

Andres Freund

Hi,

On 2022-07-21 15:26:05 +0300, Bilal Yavuz wrote:
> > On 2022-07-06 11:03:31 +0200, Peter Eisentraut wrote:
> > > dff7b5a960 meson: prereq: regress: allow to specify director containing
> > > expected files.
> > >
> > > This could use a bit more explanation, but it doesn't look
> > > controversial so far.
> 
> While testing ECPG, C and exe files are generated by meson so these files
> are in the meson's build directory but expected files are in the source
> directory. However; there was no way to set different paths for inputs (C
> and exe files') and expected files' directory. So, I added `--expecteddir`
> to separately set expected files' directory.

Attached is a version of this patch that also removes the copying of these
files from ecpg's makefile.

Bilal's version checked different directories for expected files, but I don't
think that's necessary. Bilal, do you remember why you added that?


I'm somewhat tempted to rename ecpg's pg_regress to pg_regress_ecpg as part of
this, given the .c file is named pg_regress_ecpg.c and that pg_regress is a
pre-existing binary.

Greetings,

Andres Freund

Hi,

I was looking at re-unifying gendef2.pl that the meson patchset had introduced
for temporary ease during hacking with gendef.pl. Testing that I noticed that
either I and my machine is very confused, or gendef.pl's check whether it can
skip work is bogus.

I noticed that, despite having code to avoid rerunning when the input files
are older than the .def file, it always runs.

# if the def file exists and is newer than all input object files, skip
# its creation
if (-f $deffile
    && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
{
    print "Not re-generating $defname.DEF, file already exists.\n";
    exit(0);
}

My understanding of -M is that it returns the time delta between the file
modification and the start of the script. Which makes the use of max() bogus,
since it'll return the oldest time any input has been modified, not the
newest. And the condition needs to be inverted, because we want to skip the
work if $deffile is *newer*, right?

Am I missing something here?


I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
run if there's nothing to do, and it'll e.g. not notice if there's an
additional input that wasn't there during the last invocation of gendef.pl.

Greetings,

Andres Freund

On 2022-08-09 Tu 03:10, Andres Freund wrote:
> Hi,
>
> I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> for temporary ease during hacking with gendef.pl. Testing that I noticed that
> either I and my machine is very confused, or gendef.pl's check whether it can
> skip work is bogus.
>
> I noticed that, despite having code to avoid rerunning when the input files
> are older than the .def file, it always runs.
>
> # if the def file exists and is newer than all input object files, skip
> # its creation
> if (-f $deffile
>     && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> {
>     print "Not re-generating $defname.DEF, file already exists.\n";
>     exit(0);
> }
>
> My understanding of -M is that it returns the time delta between the file
> modification and the start of the script. Which makes the use of max() bogus,
> since it'll return the oldest time any input has been modified, not the
> newest. And the condition needs to be inverted, because we want to skip the
> work if $deffile is *newer*, right?
>
> Am I missing something here?


No, you're right, this is bogus. Reversing the test and using min
instead of max is the obvious fix.


> I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> run if there's nothing to do, and it'll e.g. not notice if there's an
> additional input that wasn't there during the last invocation of gendef.pl.
>

Maybe, need to think about that more.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 8/8/22 18:53, Andres Freund wrote:
> Bilal's version checked different directories for expected files, but I don't
> think that's necessary. Bilal, do you remember why you added that?
This was for not breaking autoconf build. Autoconf wasn't using 
expecteddir, so I checked different directories.

Greetings,

Nazir Bilal Yavuz

Hi,

On 2022-06-02 10:26:09 -0700, Andres Freund wrote:
> > Could we have the meson build check that, say, if gram.c exists it
> > is newer than gram.y?  Or get it to ignore an in-tree gram.c?
>
> I suspect the problem with ignoring is gram.h, that's probably a bit harder to
> ignore.

I tried to ignore various generated files in the source tree, but I don't
think it's doable for all of them. Consider
e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
"guc-file.c" from gram.c

Because it's a "" include, the search path starts in the current directory and
only then -I is searched. To my knowledge there's no way of changing
that. Quoting the gcc manpage:

       -I dir
       -iquote dir
       -isystem dir
       -idirafter dir
           Add the directory dir to the list of directories to be searched for header files during preprocessing.  If
dirbegins with = or $SYSROOT, then
 
           the = or $SYSROOT is replaced by the sysroot prefix; see --sysroot and -isysroot.

           Directories specified with -iquote apply only to the quote form of the directive, "#include "file"".
Directoriesspecified with -I, -isystem, or
 
           -idirafter apply to lookup for both the "#include "file"" and "#include <file>" directives.

           You can specify any number or combination of these options on the command line to search for header files in
severaldirectories.  The lookup
 
           order is as follows:

           1.  For the quote form of the include directive, the directory of the current file is searched first.

           2.  For the quote form of the include directive, the directories specified by -iquote options are searched
inleft-to-right order, as they appear
 
               on the command line.

           3.  Directories specified with -I options are scanned in left-to-right order.
           [...]

Except for copying guc.c from source to build tree before building, I don't
see a way of ignoring the in-build-tree guc-file.c.

Not sure what a good way of dealing with this is. For now I'll make it just
error out if there's any known such file in the source tree, but that's not a
good solution forever.  If it were just "normal" build leftovers I'd propose
to (optionally) just remove them, but that's not good for tarballs.

Greetings,

Andres Freund

Hi,

Attached is a new version of the meson patchset. Plenty changes:

- Added a postgresql-extension.pc pkg-config file. That allows building server
  extensions without integrating directly with the postgres buildsystem. I
  have tested that this allows to build a simple out-of-tree extension on
  linux and windows - the latter is something that we didn't really support
  before.  I think we could add something similar to the autoconf build in the
  back branches, which'd make it easier to build extensions using this
  mechanism across server versions.

- A significant number of the preparatory patches has been committed

- Lots of cleanup / simplification around exporting symbols, including
  reunifying gendef.pl that I had previously copied

- Ecpg is now built and tested on windows, thanks to the above

- If there are any leftover generated files in the source tree, we now error
  out, with instructions for how to fix it. That might need a better answer at
  some point (think building from tarball), but I think that's good enough for
  now.

  It might be worth generating a file to perform the cleanups, it can be a
  long list.

- CI for Openbsd, Netbsd (thanks Bilal!), that found a few minor issues

- I hadn't fully implemented the defaults for semaphores. Turns out named
  semaphores are really slow on openbsd and netbsd.

- I went through all the "configure" tests to see if there are mismatches, and
  either fixed them or added FIXMEs. There's maybe a handful.

- The PGXS compat layer is good enough to build at least a few moderately
  complicated extensions (postgis, postgis), but currently their tests fail
  against 15 (independent of the buildsystem)...

- Improved configure summary to show CFLAGS

- Some other CI improvements, we e.g. didn't use the same test concurrency and
  CFLAGSs between the meson and autoconf tasks.

- Lots of small cleanups

- The testrunner now creates a test.start file when starting and either a
  test.success or test.failure when ending. I'd like to use that to select the
  list of log files etc to report in CI / the buildfarm, while still allowing
  concurrent testing.  Andrew, does that make sense to you?

- Lots of other small stuff


I think this is getting closer to being initially mergeable. As we'd
discussed, we're more likely to succeed if we accept working somewhat
incrementally on this.


Samay, with a bit of input from me, started on adding a docs chapter for
building with meson. I hope to include that in the next version.

I'll next send out an email discussing where test outputs should be when
running them with meson and how tests and "testsuites" should be named.

Greetings,

Andres

On Thu, Aug 11, 2022 at 12:19 AM Andres Freund <andres@anarazel.de> wrote:
> I tried to ignore various generated files in the source tree, but I don't
> think it's doable for all of them. Consider
> e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
> "guc-file.c" from gram.c

With a bit of work, we could probably get rid of those includes. See
27199058d98ef7f for one example.

-- 
John Naylor
EDB: http://www.enterprisedb.com

John Naylor <john.naylor@enterprisedb.com> writes:
> With a bit of work, we could probably get rid of those includes. See
> 27199058d98ef7f for one example.

Yeah --- it would mean creating gram.h files for all the bison grammars
not just a few of them, but it's certainly do-able if there's motivation
to make the changes.  Most of the files that are done that way date
from before we knew about flex's %top.

            regards, tom lane

I wrote:
> John Naylor <john.naylor@enterprisedb.com> writes:
>> With a bit of work, we could probably get rid of those includes. See
>> 27199058d98ef7f for one example.

> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes.  Most of the files that are done that way date
> from before we knew about flex's %top.

BTW, 72b1e3a21 is another useful precedent in this area.

            regards, tom lane

On Thu, Aug 11, 2022 at 10:37 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> John Naylor <john.naylor@enterprisedb.com> writes:
> > With a bit of work, we could probably get rid of those includes. See
> > 27199058d98ef7f for one example.
>
> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes.  Most of the files that are done that way date
> from before we knew about flex's %top.

I'll volunteer to work on this unless an easier solution happens to
come along in the next couple days. (aside: guc-file.l doesn't have a
grammar, so not yet sure if that makes the issue easier or harder...)

-- 
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-08-11 10:57:33 +0700, John Naylor wrote:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days.

Cool!


> (aside: guc-file.l doesn't have a grammar, so not yet sure if that makes the
> issue easier or harder...)

I think we should consider compiling it separately from guc.c. guc.c already
compiles quite slowly (iirc beat only by ecpg and main grammar), and it's a
relatively commonly changed source file.

It might even be a good idea to split guc.c so it only contains the settings
arrays + direct dependencies...

Greetings,

Andres Freund

John Naylor <john.naylor@enterprisedb.com> writes:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days. (aside: guc-file.l doesn't have a
> grammar, so not yet sure if that makes the issue easier or harder...)

That one's probably mostly about the issue mentioned in the other
commit you identified.  Without %top, it's impossible to make a
standalone flex module honor the rule about thou-shalt-have-no-
other-includes-before-postgres.h.  So embedding it in some other
file was originally a necessity for that.  Now that we know how
to fix that, it's just a matter of making sure that any other stuff
the scanner needs is available from a .h file.

            regards, tom lane

Starting a new thread to control clutter. [was: Re: [RFC] building
postgres with meson]

motivation: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de

On Thu, Aug 11, 2022 at 11:07 AM Andres Freund <andres@anarazel.de> wrote:
> I think we should consider compiling it separately from guc.c. guc.c already
> compiles quite slowly (iirc beat only by ecpg and main grammar), and it's a
> relatively commonly changed source file.

Done in the attached, and will do the rest in time. It seemed most
straightforward to put ProcessConfigFileInternal() in guc.c since
that's where most of its callers are, and it relies on some vars and
types declared there. There are a couple new extern declarations in
guc.h that are only for guc.c and guc-file.c:

+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);

These might be better placed in a new guc_internal.h. Thoughts?

> It might even be a good idea to split guc.c so it only contains the settings
> arrays + direct dependencies...

Perhaps this can be a TODO item, one which falls under "[E] marks
items that are easier to implement". I've been slacking on removing
the old/intractable cruft from the TODO list, but we should also be
sticking small nice-but-not-necessary things in there. That said, if
this idea has any bearing on the guc_internal.h idea, it might be
better dealt with now.

-- 
John Naylor
EDB: http://www.enterprisedb.com

Here are the rest. Most of it was pretty straightforward, with the
main exception of jsonpath_scan.c, which is not quite finished. That
one passes tests but still has one compiler warning. I'm unsure how
much of what is there already is really necessary or was cargo-culted
from elsewhere without explanation. For starters, I'm not sure why the
grammar has a forward declaration of "union YYSTYPE". It's noteworthy
that it used to compile standalone, but with a bit more stuff, and
that was reverted in 550b9d26f80fa30. I can hack on it some more later
but I ran out of steam today.

Other questions thus far:

- "BISONFLAGS += -d" is now in every make file with a .y file -- can
we just force that everywhere?

- Include order seems to matter for the grammar's .h file. I didn't
test if that was the case every time, and after a few miscompiles just
always made it the last inclusion, but I'm wondering if we should keep
those inclusions outside %top{} and put it at the start of the next
%{} ?

- contrib/cubeparse.y now has a global variable -- not terrific, but I
wanted to get something working first.

- I'm actually okay with guc-file.c now, but I'll still welcome
comments on that.

-- 
John Naylor
EDB: http://www.enterprisedb.com

Hi,

Thanks for your work on this!

On 2022-08-13 15:39:06 +0700, John Naylor wrote:
> Here are the rest. Most of it was pretty straightforward, with the
> main exception of jsonpath_scan.c, which is not quite finished. That
> one passes tests but still has one compiler warning. I'm unsure how
> much of what is there already is really necessary or was cargo-culted
> from elsewhere without explanation. For starters, I'm not sure why the
> grammar has a forward declaration of "union YYSTYPE". It's noteworthy
> that it used to compile standalone, but with a bit more stuff, and
> that was reverted in 550b9d26f80fa30. I can hack on it some more later
> but I ran out of steam today.

I'm not sure either...


> Other questions thus far:
> 
> - "BISONFLAGS += -d" is now in every make file with a .y file -- can
> we just force that everywhere?

Hm. Not sure it's worth it, extensions might use our BISON stuff...


> - Include order seems to matter for the grammar's .h file. I didn't
> test if that was the case every time, and after a few miscompiles just
> always made it the last inclusion, but I'm wondering if we should keep
> those inclusions outside %top{} and put it at the start of the next
> %{} ?

I think we have a few of those dependencies already, see e.g.
/*
 * NB: include gram.h only AFTER including scanner.h, because scanner.h
 * is what #defines YYLTYPE.
 */


> From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
> From: John Naylor <john.naylor@postgresql.org>
> Date: Thu, 11 Aug 2022 19:38:37 +0700
> Subject: [PATCH v201 1/9] Build guc-file.c standalone

Might be worth doing some of the moving around here separately from the
parser/scanner specific bits.


> +/* functions shared between guc.c and guc-file.l */
> +extern int    guc_name_compare(const char *namea, const char *nameb);
> +extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
> +                                                 bool applySettings, int elevel);
> +extern void record_config_file_error(const char *errmsg,
> +                                     const char *config_file,
> +                                     int lineno,
> +                                     ConfigVariable **head_p,
> +                                     ConfigVariable **tail_p);
>  
>  /*
>   * The following functions are not in guc.c, but are declared here to avoid
> -- 
> 2.36.1
> 

I think I prefer your suggestion of a guc_internal.h upthread.



> From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> From: John Naylor <john.naylor@postgresql.org>
> Date: Fri, 12 Aug 2022 15:45:24 +0700
> Subject: [PATCH v201 2/9] Build booscanner.c standalone

> -# bootscanner is compiled as part of bootparse
> -bootparse.o: bootscanner.c
> +# See notes in src/backend/parser/Makefile about the following two rules
> +bootparse.h: bootparse.c
> +    touch $@
> +
> +bootparse.c: BISONFLAGS += -d
> +
> +# Force these dependencies to be known even without dependency info built:
> +bootparse.o bootscan.o: bootparse.h

Wonder if we could / should wrap this is something common. It's somewhat
annoying to repeat this stuff everywhere.



> diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
> index aa6e89268e..2dc292c21d 100644
> --- a/src/test/isolation/specscanner.l
> +++ b/src/test/isolation/specscanner.l
> @@ -1,4 +1,4 @@
> -%{
> +%top{
>  /*-------------------------------------------------------------------------
>   *
>   * specscanner.l
> @@ -9,7 +9,14 @@
>   *
>   *-------------------------------------------------------------------------
>   */
> +#include "postgres_fe.h"

Miniscule nitpick: I think we typically leave an empty line between header and
first include.


> diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> index dbe7d4f742..0b373048b5 100644
> --- a/contrib/cube/cubedata.h
> +++ b/contrib/cube/cubedata.h
> @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
>  
>  /* in cubeparse.y */
>  extern int    cube_yyparse(NDBOX **result);
> +
> +/* All grammar constructs return strings */
> +#define YYSTYPE char *

Why does this need to be defined in a semi-public header? If we do this in
multiple files we'll end up with the danger of macro redefinition warnings.


> +extern int scanbuflen;

The code around scanbuflen seems pretty darn grotty. Allocating enough memory
for the entire list by allocating the entire string size... I don't know
anything about contrib/cube, but isn't that in effect O(inputlen^2) memory?


Greetings,

Andres Freund

For v3, I addressed some comments and added .h files to the
headerscheck exceptions.

On Tue, Aug 16, 2022 at 1:11 AM Andres Freund <andres@anarazel.de> wrote:

> On 2022-08-13 15:39:06 +0700, John Naylor wrote:
> > Here are the rest. Most of it was pretty straightforward, with the
> > main exception of jsonpath_scan.c, which is not quite finished. That
> > one passes tests but still has one compiler warning. I'm unsure how
> > much of what is there already is really necessary or was cargo-culted
> > from elsewhere without explanation. For starters, I'm not sure why the
> > grammar has a forward declaration of "union YYSTYPE". It's noteworthy
> > that it used to compile standalone, but with a bit more stuff, and
> > that was reverted in 550b9d26f80fa30. I can hack on it some more later
> > but I ran out of steam today.

I've got it in half-way decent shape now, with an *internal.h header
and some cleanups.

> > - Include order seems to matter for the grammar's .h file. I didn't
> > test if that was the case every time, and after a few miscompiles just
> > always made it the last inclusion, but I'm wondering if we should keep
> > those inclusions outside %top{} and put it at the start of the next
> > %{} ?
>
> I think we have a few of those dependencies already, see e.g.
> /*
>  * NB: include gram.h only AFTER including scanner.h, because scanner.h
>  * is what #defines YYLTYPE.
>  */

Went with something like this in all cases:

/*
 * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
 * includes node definitions needed for YYSTYPE.
 */

Future cleanup: I see this in headerscheck:

# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.

That directive has been supported in Bison since 2.4.2.

> > From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
> > From: John Naylor <john.naylor@postgresql.org>
> > Date: Thu, 11 Aug 2022 19:38:37 +0700
> > Subject: [PATCH v201 1/9] Build guc-file.c standalone
>
> Might be worth doing some of the moving around here separately from the
> parser/scanner specific bits.

Done in 0001/0003.

> > +/* functions shared between guc.c and guc-file.l */
> > [...]
> I think I prefer your suggestion of a guc_internal.h upthread.

Started in 0002, but left open the headerscheck failure.

Also, if such a thing is meant to be #include'd only by two generated
files, maybe it should just live in the directory where they live, and
not in the src/include dir?

> > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > From: John Naylor <john.naylor@postgresql.org>
> > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > Subject: [PATCH v201 2/9] Build booscanner.c standalone
>
> > -# bootscanner is compiled as part of bootparse
> > -bootparse.o: bootscanner.c
> > +# See notes in src/backend/parser/Makefile about the following two rules
> > +bootparse.h: bootparse.c
> > +     touch $@
> > +
> > +bootparse.c: BISONFLAGS += -d
> > +
> > +# Force these dependencies to be known even without dependency info built:
> > +bootparse.o bootscan.o: bootparse.h
>
> Wonder if we could / should wrap this is something common. It's somewhat
> annoying to repeat this stuff everywhere.

I haven't looked at the Meson effort recently, but if the build rule
is less annoying there, I'm inclined to leave this as a wart until
autotools are retired.

> > diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
> > index aa6e89268e..2dc292c21d 100644
> > --- a/src/test/isolation/specscanner.l
> > +++ b/src/test/isolation/specscanner.l
> > @@ -1,4 +1,4 @@
> > -%{
> > +%top{
> >  /*-------------------------------------------------------------------------
> >   *
> >   * specscanner.l
> > @@ -9,7 +9,14 @@
> >   *
> >   *-------------------------------------------------------------------------
> >   */
> > +#include "postgres_fe.h"
>
> Miniscule nitpick: I think we typically leave an empty line between header and
> first include.

In a small unscientific sample it seems like the opposite is true
actually, but I'll at least try to be consistent within the patch set.

> > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > index dbe7d4f742..0b373048b5 100644
> > --- a/contrib/cube/cubedata.h
> > +++ b/contrib/cube/cubedata.h
> > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> >
> >  /* in cubeparse.y */
> >  extern int   cube_yyparse(NDBOX **result);
> > +
> > +/* All grammar constructs return strings */
> > +#define YYSTYPE char *
>
> Why does this need to be defined in a semi-public header? If we do this in
> multiple files we'll end up with the danger of macro redefinition warnings.

I tried to put all the Flex/Bison stuff in another *_internal header,
but that breaks the build. Putting just this one symbol in a header is
silly, but done that way for now. Maybe two copies of the symbol?

Another future cleanup: "%define api.prefix {cube_yy}" etc would cause
it to be spelled CUBE_YYSTYPE (other macros too), sidestepping this
problem (requires Bison 2.6). IIUC, doing it our way has been
deprecated for 9 years.

> > +extern int scanbuflen;
>
> The code around scanbuflen seems pretty darn grotty. Allocating enough memory
> for the entire list by allocating the entire string size... I don't know
> anything about contrib/cube, but isn't that in effect O(inputlen^2) memory?

Neither do I.




--
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-08-16 17:41:43 +0700, John Naylor wrote:
> For v3, I addressed some comments and added .h files to the
> headerscheck exceptions.

Thanks!


> /*
>  * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
>  * includes node definitions needed for YYSTYPE.
>  */
> 
> Future cleanup: I see this in headerscheck:
> 
> # We can't make these Bison output files compilable standalone
> # without using "%code require", which old Bison versions lack.
> # parser/gram.h will be included by parser/gramparse.h anyway.
> 
> That directive has been supported in Bison since 2.4.2.

2.4.2 is from 2010. So I think we could just start relying on it?


> > > +/* functions shared between guc.c and guc-file.l */
> > > [...]
> > I think I prefer your suggestion of a guc_internal.h upthread.
> 
> Started in 0002, but left open the headerscheck failure.
> 
> Also, if such a thing is meant to be #include'd only by two generated
> files, maybe it should just live in the directory where they live, and
> not in the src/include dir?

It's not something we've done for the backend afaics, but I don't see a reason
not to start at some point.


> > > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > > From: John Naylor <john.naylor@postgresql.org>
> > > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > > Subject: [PATCH v201 2/9] Build booscanner.c standalone
> >
> > > -# bootscanner is compiled as part of bootparse
> > > -bootparse.o: bootscanner.c
> > > +# See notes in src/backend/parser/Makefile about the following two rules
> > > +bootparse.h: bootparse.c
> > > +     touch $@
> > > +
> > > +bootparse.c: BISONFLAGS += -d
> > > +
> > > +# Force these dependencies to be known even without dependency info built:
> > > +bootparse.o bootscan.o: bootparse.h
> >
> > Wonder if we could / should wrap this is something common. It's somewhat
> > annoying to repeat this stuff everywhere.
> 
> I haven't looked at the Meson effort recently, but if the build rule
> is less annoying there, I'm inclined to leave this as a wart until
> autotools are retired.

The only complicating thing in the rules there is the dependencies from one .c
file to another .c file.


> > > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > > index dbe7d4f742..0b373048b5 100644
> > > --- a/contrib/cube/cubedata.h
> > > +++ b/contrib/cube/cubedata.h
> > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > >
> > >  /* in cubeparse.y */
> > >  extern int   cube_yyparse(NDBOX **result);
> > > +
> > > +/* All grammar constructs return strings */
> > > +#define YYSTYPE char *
> >
> > Why does this need to be defined in a semi-public header? If we do this in
> > multiple files we'll end up with the danger of macro redefinition warnings.
> 
> I tried to put all the Flex/Bison stuff in another *_internal header,
> but that breaks the build. Putting just this one symbol in a header is
> silly, but done that way for now. Maybe two copies of the symbol?

The problem is that if it's in a header you can't include another header with
such a define. That's fine if it's a .h that's just intended to be included by
a limited set of files, but for something like a header for a datatype that
might need to be included to e.g. define a PL transform or a new operator or
...  This would be solved by the %code requires thing, right?


Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-08-16 17:41:43 +0700, John Naylor wrote:
>> That directive has been supported in Bison since 2.4.2.

> 2.4.2 is from 2010. So I think we could just start relying on it?

Apple is still shipping 2.3.  Is this worth enough to make Mac
users install a non-default Bison?  I seriously doubt it.

I don't say that there won't be a reason that justifies that
at some point, but letting headerscheck test autogenerated
files seems of only microscopic benefit :-(

            regards, tom lane

On Wed, Aug 17, 2022 at 8:14 AM Andres Freund <andres@anarazel.de> wrote:

> > > > +/* functions shared between guc.c and guc-file.l */
> > > > [...]
> > > I think I prefer your suggestion of a guc_internal.h upthread.
> >
> > Started in 0002, but left open the headerscheck failure.
> >
> > Also, if such a thing is meant to be #include'd only by two generated
> > files, maybe it should just live in the directory where they live, and
> > not in the src/include dir?
>
> It's not something we've done for the backend afaics, but I don't see a reason
> not to start at some point.

BTW, I forgot to mention I did this for the json path parser, which
makes the makefile code simpler than what was there before
550b9d26f80fa30. AFAICS, we could also do the same for gramparse.h,
which is internal to parser.c. If I'm not mistaken, the only reason we
symlink gram.h to src/include/* is so that gramparse.h can include it.
So keeping gramparse.h in the backend could allow removing some gram.h
makefile incantations.

> > > Why does this need to be defined in a semi-public header? If we do this in
> > > multiple files we'll end up with the danger of macro redefinition warnings.
> >
> > I tried to put all the Flex/Bison stuff in another *_internal header,
> > but that breaks the build. Putting just this one symbol in a header is
> > silly, but done that way for now. Maybe two copies of the symbol?
>
> The problem is that if it's in a header you can't include another header with
> such a define. That's fine if it's a .h that's just intended to be included by
> a limited set of files, but for something like a header for a datatype that
> might need to be included to e.g. define a PL transform or a new operator or
> ...  This would be solved by the %code requires thing, right?

I believe it would.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On 11.08.22 02:20, Andres Freund wrote:
> Attached is a new version of the meson patchset. Plenty changes:

I have various bits of comments on this.

- There are various references to "pch" (pre-compiled headers).  Is
   there more discussion anywhere about this?  I don't know what this
   would entail or whether there are any drawbacks to be aware of.  The
   new *_pch.h files don't have any comments.  Maybe this should be a
   separate patch later.

- About relativize_shared_library_references: We have had several
   patches over the years for working around SIP stuff, and some of
   them did essentially this, but we decided not to go ahead with them.
   We could revisit that, but it should be a separate patch, not mixed
   in with this.

- postgresql-extension.pc: Similarly, this ought to be a separate
   patch.  If we want people to use this, we'll need it in the makefile
   build system anyway.

- -DFRONTEND is used somewhat differently from the makefiles.  For
    example, meson sets -DFRONTEND for pg_controldata, but the
    makefiles don't.  Conversely, the makefiles set -DFRONTEND for
    ecpglib, but meson does not.  This should be checked again to make
    sure it all matches up.

- Option name spelling should be make consistent about underscores
   versus hyphens.  Built-in meson options use underscores, so we
   should make the user-defined ones like that as well (some already
   do).  (wal-blocksize krb-srvnam system-tzdata tap-tests bsd-auth)

- I have found the variable name "cdata" for configuration_data() to
   be less than clear.  I see some GNOME projects to it that way, is
   that where it's from?  systemd uses "conf", maybe that's better.

- In the top-level meson.build, the "renaming" of the Windows system
   name

       host_system = host_machine.system() == 'windows' ? 'win32' : 
host_machine.system()
       build_system = build_machine.system() == 'windows' ? 'win32' : 
build_machine.system()

   seems unnecessary to me.  Why not stick with the provided names?

- The c99_test ought to be not needed if the c_std project option is
   used.  Was there a problem with that?

- Is there a way to split up the top-level meson.build somehow?  Maybe
   just throw some stuff into included files?  This might get out of
   hand at some point.

- The PG_SYSROOT detection gives different results.  On my system,
   configure produces
 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk,
   meson produces
 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk.
   src/template/darwin goes out of its way to get a version-specific
   result, so we need to carry that over somehow.  (The difference does
   result in differences in the built binaries.)


Then, some patches from me:

0001-Change-shared-library-installation-naming-on-macOS.patch

This changes the makefiles to make the shared library file naming on
macOS match what meson produces.  I don't know what the situation is
on other platforms.

0002-meson-Fix-installation-name-of-libpgfeutils.patch

This was presumably an accidental mistake.

0003-meson-Libraries-need-to-be-built-with-DSO_MAJOR_VERS.patch

This is needed to make NLS work for the libraries.

0004-meson-Add-darwin_versions-argument-for-libraries.patch

This is to make the output match what Makefile.shlib produces.

0005-meson-Fix-link-order-of-support-libraries.patch
0006-meson-Make-link-order-of-external-libraries-match-ma.patch
0007-WIP-meson-Make-link-order-of-object-files-match-make.patch

I have analyzed the produced binaries between both build systems to
make sure they match.  If we link the files and libraries in different
orders, that becomes difficult.  So this fixes this up a bit.  0005 is
needed for correctness in general, I think.  0006 is mostly cosmetic.
You probably wanted to make the library order alphabetical in the
meson files, which I'd support, but then we should change the
makefiles to match.  Similarly, 0007, which is clearly a bit messy at
the moment, but we should try to sort that out either in the old or
the new build files.


And finally some comments on your patches:

meson: prereq: Don't add HAVE_LDAP_H HAVE_WINLDAP_H to pg_config.h.

This can go ahead.

meson: prereq: fix warning compat_informix/rnull.pgc with msvc

-   $float f = 3.71;
+   $float f = (float) 3.71;

This could use float literals like

+   $float f = 3.71f;

Hi,

On 2022-08-17 15:50:23 +0200, Peter Eisentraut wrote:
> - There are various references to "pch" (pre-compiled headers).  Is
>   there more discussion anywhere about this?  I don't know what this
>   would entail or whether there are any drawbacks to be aware of.  The
>   new *_pch.h files don't have any comments.  Maybe this should be a
>   separate patch later.

It's mainly to make windows builds a bit slower. I've no objection to
separating this out.


> - About relativize_shared_library_references: We have had several
>   patches over the years for working around SIP stuff, and some of
>   them did essentially this, but we decided not to go ahead with them.
>   We could revisit that, but it should be a separate patch, not mixed
>   in with this.

The prior approaches all had issues because they didn't support relative
references IIRC (and thus broke being able to relocate the installation),
which this does.

I just found it very annoying to work on macs without this. And there were at
least two "bug" reports of testers of the meson branch that were just due to
SIP.

I'm ok with splitting it out, but I also think it's a lower risk opportunity
to test that this works.


> - postgresql-extension.pc: Similarly, this ought to be a separate
>   patch.  If we want people to use this, we'll need it in the makefile
>   build system anyway.

Makes sense. I'd like to keep it in the same patch for a short while longer,
to deduplicate some of the code, but then will split it out.


> - -DFRONTEND is used somewhat differently from the makefiles.  For
>    example, meson sets -DFRONTEND for pg_controldata, but the
>    makefiles don't.  Conversely, the makefiles set -DFRONTEND for
>    ecpglib, but meson does not.  This should be checked again to make
>    sure it all matches up.

Yes, should sync that up.

FWIW, meson does add -DFRONTEND for ecpglib. There were a few places that did
add it twice, I'll push a cleanup of that in a bit.


> - Option name spelling should be make consistent about underscores
>   versus hyphens.  Built-in meson options use underscores, so we
>   should make the user-defined ones like that as well (some already
>   do).  (wal-blocksize krb-srvnam system-tzdata tap-tests bsd-auth)

No objection.


> - I have found the variable name "cdata" for configuration_data() to
>   be less than clear.  I see some GNOME projects to it that way, is
>   that where it's from?  systemd uses "conf", maybe that's better.

I don't know where it's from - I don't think I ever looked at gnome
buildsystem stuff. It seems to be the obvious abbreviation for
configuration_data()...  I don't object to conf, but it's not a clear
improvement to me.


> - In the top-level meson.build, the "renaming" of the Windows system
>   name
>
>       host_system = host_machine.system() == 'windows' ? 'win32' :
> host_machine.system()
>       build_system = build_machine.system() == 'windows' ? 'win32' :
> build_machine.system()
>
>   seems unnecessary to me.  Why not stick with the provided names?

Because right now we also use it for things like choosing the "source" for
pg_config_os.h (i.e. include/port/{darwin,linux,win32,..}.h). And it seemed
easier to just have one variable name for all of it.


> - The c99_test ought to be not needed if the c_std project option is
>   used.  Was there a problem with that?

We don't want to force -std=c99 when not necessary, I think. We sometimes use
features from newer (and from gnu) language versions after testing
availability, and if we hardcode the version those will either fail or elicit
warnings.


> - Is there a way to split up the top-level meson.build somehow?  Maybe
>   just throw some stuff into included files?  This might get out of
>   hand at some point.

We can put stuff into config/meson.build or such. But I don't think it's
clearly warranted at this point.


> - The PG_SYSROOT detection gives different results.  On my system,
>   configure produces
>
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk,
>   meson produces
>
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk.
>   src/template/darwin goes out of its way to get a version-specific
>   result, so we need to carry that over somehow.  (The difference does
>   result in differences in the built binaries.)

TBH, I don't really understand the SYSROOT stuff all that well, never having
used a mac in anger (well, only in anger, but ...).

What do you think about extracting the relevant portion of src/template/darwin
into a dedicated shell script that gets called by both?


> Then, some patches from me:
>
> 0001-Change-shared-library-installation-naming-on-macOS.patch
>
> This changes the makefiles to make the shared library file naming on
> macOS match what meson produces.  I don't know what the situation is
> on other platforms.

No opinion on the matter. Seems best to apply separately if we want to?


> 0002-meson-Fix-installation-name-of-libpgfeutils.patch
>
> This was presumably an accidental mistake.

Yes, merged.


> 0003-meson-Libraries-need-to-be-built-with-DSO_MAJOR_VERS.patch
>
> This is needed to make NLS work for the libraries.

Oh, huh. Yes, merged.


> 0004-meson-Add-darwin_versions-argument-for-libraries.patch
>
> This is to make the output match what Makefile.shlib produces.

:/, merged. Would be good to clean up at some point.


> 0005-meson-Fix-link-order-of-support-libraries.patch
> 0006-meson-Make-link-order-of-external-libraries-match-ma.patch
> 0007-WIP-meson-Make-link-order-of-object-files-match-make.patch
>
> I have analyzed the produced binaries between both build systems to
> make sure they match.  If we link the files and libraries in different
> orders, that becomes difficult.  So this fixes this up a bit.  0005 is
> needed for correctness in general, I think.

Makes sense.


> 0006 is mostly cosmetic.  You probably wanted to make the library order
> alphabetical in the meson files, which I'd support, but then we should
> change the makefiles to match.

Trying to match makefile order doesn't seem like a good plan, given that it's
effectively random, and might change depending on dependencies of linked to
libraries etc.

Isn't the use of AC_CHECK_LIB for at least lz4, zstd completely bogus? We get
whether they're available via pkg-config, but then completely ignore the
linker flag for the library name. The comment says:
  # We only care about -I, -D, and -L switches;
  # note that -llz4 will be added by AC_CHECK_LIB below.
but without any further explanation. This seems to be from 4d399a6fbeb.


The repetition of lz4, zstd in pg_rewind, pg_waldump and backend makes me
wonder if we should put them in a xlogreader_deps or such. It's otherwise not
obvious why pg_rewind, pg_waldump need lz4/zstd.


> Similarly, 0007, which is clearly a bit
> messy at the moment, but we should try to sort that out either in the old or
> the new build files.

I am against trying to maintain bug-for-bug compatibility on filename
ordering. But obviously ok with fixing the ordering to make sense on both
sides.

What was your decision point about when to adjust makefile ordering and when
meson ordering?



> And finally some comments on your patches:

Any comment on the pg_regress_ecpg commit? I'd like to get that out of the
way, and it seems considerably cleaner than the hackery we do right now to
make VPATH builds work.


> meson: prereq: Don't add HAVE_LDAP_H HAVE_WINLDAP_H to pg_config.h.
>
> This can go ahead.
>
> meson: prereq: fix warning compat_informix/rnull.pgc with msvc
>
> -   $float f = 3.71;
> +   $float f = (float) 3.71;
>
> This could use float literals like
>
> +   $float f = 3.71f;

I tried that first, but it fails:
../src/interfaces/ecpg/test/compat_informix/rnull.pgc:19: ERROR: trailing junk after numeric literal

Should have noted that. I don't feel like fixing ecpg's parser etc...


Greetings,

Andres Freund

On Wed, Aug 17, 2022 at 8:14 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-08-16 17:41:43 +0700, John Naylor wrote:
> > For v3, I addressed some comments and added .h files to the
> > headerscheck exceptions.
>
> Thanks!
>
>
> > /*
> >  * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
> >  * includes node definitions needed for YYSTYPE.
> >  */
> >
> > Future cleanup: I see this in headerscheck:
> >
> > # We can't make these Bison output files compilable standalone
> > # without using "%code require", which old Bison versions lack.
> > # parser/gram.h will be included by parser/gramparse.h anyway.
> >
> > That directive has been supported in Bison since 2.4.2.
>
> 2.4.2 is from 2010. So I think we could just start relying on it?
>
>
> > > > +/* functions shared between guc.c and guc-file.l */
> > > > [...]
> > > I think I prefer your suggestion of a guc_internal.h upthread.
> >
> > Started in 0002, but left open the headerscheck failure.
> >
> > Also, if such a thing is meant to be #include'd only by two generated
> > files, maybe it should just live in the directory where they live, and
> > not in the src/include dir?
>
> It's not something we've done for the backend afaics, but I don't see a reason
> not to start at some point.
>
>
> > > > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > > > From: John Naylor <john.naylor@postgresql.org>
> > > > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > > > Subject: [PATCH v201 2/9] Build booscanner.c standalone
> > >
> > > > -# bootscanner is compiled as part of bootparse
> > > > -bootparse.o: bootscanner.c
> > > > +# See notes in src/backend/parser/Makefile about the following two rules
> > > > +bootparse.h: bootparse.c
> > > > +     touch $@
> > > > +
> > > > +bootparse.c: BISONFLAGS += -d
> > > > +
> > > > +# Force these dependencies to be known even without dependency info built:
> > > > +bootparse.o bootscan.o: bootparse.h
> > >
> > > Wonder if we could / should wrap this is something common. It's somewhat
> > > annoying to repeat this stuff everywhere.
> >
> > I haven't looked at the Meson effort recently, but if the build rule
> > is less annoying there, I'm inclined to leave this as a wart until
> > autotools are retired.
>
> The only complicating thing in the rules there is the dependencies from one .c
> file to another .c file.
>
>
> > > > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > > > index dbe7d4f742..0b373048b5 100644
> > > > --- a/contrib/cube/cubedata.h
> > > > +++ b/contrib/cube/cubedata.h
> > > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > > >
> > > >  /* in cubeparse.y */
> > > >  extern int   cube_yyparse(NDBOX **result);
> > > > +
> > > > +/* All grammar constructs return strings */
> > > > +#define YYSTYPE char *
> > >
> > > Why does this need to be defined in a semi-public header? If we do this in
> > > multiple files we'll end up with the danger of macro redefinition warnings.

For v4, I #defined YYSTYPE

-- 
John Naylor
EDB: http://www.enterprisedb.com

> > > > > index dbe7d4f742..0b373048b5 100644
> > > > > --- a/contrib/cube/cubedata.h
> > > > > +++ b/contrib/cube/cubedata.h
> > > > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > > > >
> > > > >  /* in cubeparse.y */
> > > > >  extern int   cube_yyparse(NDBOX **result);
> > > > > +
> > > > > +/* All grammar constructs return strings */
> > > > > +#define YYSTYPE char *
> > > >
> > > > Why does this need to be defined in a semi-public header? If we do this in
> > > > multiple files we'll end up with the danger of macro redefinition warnings.
>
> For v4, I #defined YYSTYPE

Sorry for the misfire. Continuing on, I #defined YYSTYPE in cubescan.l
before #including cubeparse.h.

I also added scanbuflen to the %parse-param to prevent resorting to a
global variable. The rest of the patches are unchanged.

-- 
John Naylor
EDB: http://www.enterprisedb.com

I wrote
> [v4]

This piece is a leftover from the last version, and forgot to remove
it, will fix:

diff --git a/contrib/cube/cubeparse.y b/contrib/cube/cubeparse.y
index 7577c4515c..e3b750b695 100644
--- a/contrib/cube/cubeparse.y
+++ b/contrib/cube/cubeparse.y
@@ -7,6 +7,7 @@
 #include "postgres.h"

 #include "cubedata.h"
+#include "cube_internal.h"
 #include "utils/float.h"

-- 
John Naylor
EDB: http://www.enterprisedb.com

On 17.08.22 23:53, Andres Freund wrote:
> Any comment on the pg_regress_ecpg commit? I'd like to get that out of the
> way, and it seems considerably cleaner than the hackery we do right now to
> make VPATH builds work.

That one looks like a very good improvement.

Hi,

On 2022-08-20 09:38:48 +0200, Peter Eisentraut wrote:
> On 17.08.22 23:53, Andres Freund wrote:
> > Any comment on the pg_regress_ecpg commit? I'd like to get that out of the
> > way, and it seems considerably cleaner than the hackery we do right now to
> > make VPATH builds work.
> 
> That one looks like a very good improvement.

Thanks for checking! Pushed.

Greetings,

Andres Freund

Hi,

On 2022-08-09 08:37:16 -0400, Andrew Dunstan wrote:
> On 2022-08-09 Tu 03:10, Andres Freund wrote:
> > Hi,
> >
> > I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> > for temporary ease during hacking with gendef.pl. Testing that I noticed that
> > either I and my machine is very confused, or gendef.pl's check whether it can
> > skip work is bogus.
> >
> > I noticed that, despite having code to avoid rerunning when the input files
> > are older than the .def file, it always runs.
> >
> > # if the def file exists and is newer than all input object files, skip
> > # its creation
> > if (-f $deffile
> >     && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> > {
> >     print "Not re-generating $defname.DEF, file already exists.\n";
> >     exit(0);
> > }
> >
> > My understanding of -M is that it returns the time delta between the file
> > modification and the start of the script. Which makes the use of max() bogus,
> > since it'll return the oldest time any input has been modified, not the
> > newest. And the condition needs to be inverted, because we want to skip the
> > work if $deffile is *newer*, right?
> >
> > Am I missing something here?
> 
> 
> No, you're right, this is bogus. Reversing the test and using min
> instead of max is the obvious fix.
> 
> 
> > I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> > run if there's nothing to do, and it'll e.g. not notice if there's an
> > additional input that wasn't there during the last invocation of gendef.pl.
> >
> 
> Maybe, need to think about that more.

Any thoughts?

I'd like to commit 0003 in
https://postgr.es/m/20220811002012.ju3rrz47i2e5tdha%40awork3.anarazel.de
fairly soon.

I did fix the bogus "die" message I added during some debugging since posting
that...

Greetings,

Andres Freund

I have looked at your branch at 0545eec895:

258f6dc0a7 Don't hardcode tmp_check/ as test directory for tap tests
8ecc33cf04 Split TESTDIR into TESTLOGDIR and TESTDATADIR

I think these patches are split up a bit incorrectly.  If you apply
the first patch by itself, then the output appears in tab_comp_dir/
directly under the source directory.  And then the second patch moves
it to tmp_check/tap_comp_dir/.  If there is an intent to apply these
patches separately somehow, this should be cleaned up.

I haven't checked the second patch in detail yet, but it looks like
the thought was that the first patch is about ready to go.

834a40e609 meson: prereq: Extend gendef.pl in preparation for meson

I'm not qualified to check that in detail, but it looks reasonable
enough to me.

See attached patch (0001) for a perlcritic fix.

97a0b096e8 meson: prereq: Add src/tools/gen_export.pl

This produces leading whitespace in the output files that at least on
darwin wasn't there before.  See attached patch (0002).  This should
be checked again on other platforms as well.

Other than that this looks good.  Attached is a small cosmetic patch (0003).

40e363b263 meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build

Since I last looked, this has been turned into a meson option.  Which
is probably the best solution.  But then we should probably make this
a configure option as well.  Otherwise, it could get a bit confusing.
For example, I just unset PG_TEST_EXTRA in my environment to test
something with the meson build, but I was unaware that meson captures
the value at setup time, so my unsetting had no effect.

In any case, maybe adjust the regular expressions to check for word
boundaries, to maintain the original "whitespace-separated"
specification.  For example,

     elsif ($ENV{PG_TEST_EXTRA} !~ /\bssl\b/)

e0a8387660 solaris: Use versioning scripts instead of -Bsymbolic

This looks like a good idea.  The documentation clearly states that
-Bsymbolic shouldn't be used, at least not in the way we have been
doing.  Might as well go ahead with this and give it a whirl on the
build farm.

0545eec895 meson: Add docs

We should think more about how to arrange the documentation.  We
probably don't want to copy-and-paste all the introductory and
requirements information.  I think we can make this initially much
briefer, like the Windows installation chapter.  For example, instead
of documenting each setup option again, just mention which ones exist
and then point (link) to the configure chapter for details.


I spent a bit of time with the test suites.  I think there is a
problem in that selecting a test suite directly, like

     meson test -C _build --suite recovery

doesn't update the tmp_install.  So if this is the first thing you run
after a build, everything will fail.  Also, if you run this later, the
tmp_install doesn't get updated, so you're not testing up-to-date
code.

Hi,

On 2022-08-24 11:39:06 +0200, Peter Eisentraut wrote:
> I have looked at your branch at 0545eec895:
> 
> 258f6dc0a7 Don't hardcode tmp_check/ as test directory for tap tests
> 8ecc33cf04 Split TESTDIR into TESTLOGDIR and TESTDATADIR
> 
> I think these patches are split up a bit incorrectly.  If you apply
> the first patch by itself, then the output appears in tab_comp_dir/
> directly under the source directory.  And then the second patch moves
> it to tmp_check/tap_comp_dir/.  If there is an intent to apply these
> patches separately somehow, this should be cleaned up.

How is that happening with that version of the patch? The test puts
tap_comp_dir under TESTDIR, and TESTDIR is $(CURDIR)/tmp_check. There was an
earlier version of the patch that was split one more time that did have that
problem, but I don't quite see how that version has it?


> I haven't checked the second patch in detail yet, but it looks like
> the thought was that the first patch is about ready to go.
> 
> 834a40e609 meson: prereq: Extend gendef.pl in preparation for meson
> 
> I'm not qualified to check that in detail, but it looks reasonable
> enough to me.
> 
> See attached patch (0001) for a perlcritic fix.

Thanks.


> 97a0b096e8 meson: prereq: Add src/tools/gen_export.pl
> 
> This produces leading whitespace in the output files that at least on
> darwin wasn't there before.  See attached patch (0002).  This should
> be checked again on other platforms as well.

Hm, to me the indentation as is makes more sense, but ...

> Other than that this looks good.  Attached is a small cosmetic patch (0003).

I wonder if we should rewrite this in python - I chose perl because I thought
we could share it, but as you pointed out, that's not possible, because we
don't want to depend on perl during the autoconf build from a tarball.


> e0a8387660 solaris: Use versioning scripts instead of -Bsymbolic
> 
> This looks like a good idea.  The documentation clearly states that
> -Bsymbolic shouldn't be used, at least not in the way we have been
> doing.  Might as well go ahead with this and give it a whirl on the
> build farm.

Cool. I looked at this because I was confused about getting warnings with
autoconf that I wasn't getting with meson.


> 0545eec895 meson: Add docs
> 
> We should think more about how to arrange the documentation.  We
> probably don't want to copy-and-paste all the introductory and
> requirements information.  I think we can make this initially much
> briefer, like the Windows installation chapter.  For example, instead
> of documenting each setup option again, just mention which ones exist
> and then point (link) to the configure chapter for details.

The current docs, including the windows ones, are already hard to follow. I
think we should take some care to not make the meson bits even more
confusing. Cross referencing left and right seems problematic from that angle.


> I spent a bit of time with the test suites.  I think there is a
> problem in that selecting a test suite directly, like
> 
>     meson test -C _build --suite recovery
> 
> doesn't update the tmp_install.  So if this is the first thing you run
> after a build, everything will fail.  Also, if you run this later, the
> tmp_install doesn't get updated, so you're not testing up-to-date
> code.

At the moment creation of the tmp_install is its own test suite. I don't know
if that's the best way, or what the best way is, but that explains that
fact. You can do the above without the issue by specifying
--suite setup --suite recovery.


Greetings,

Andres Freund

Hi,

On 2022-08-17 14:53:17 -0700, Andres Freund wrote:
> > - In the top-level meson.build, the "renaming" of the Windows system
> >   name
> >
> >       host_system = host_machine.system() == 'windows' ? 'win32' :
> > host_machine.system()
> >       build_system = build_machine.system() == 'windows' ? 'win32' :
> > build_machine.system()
> >
> >   seems unnecessary to me.  Why not stick with the provided names?
> 
> Because right now we also use it for things like choosing the "source" for
> pg_config_os.h (i.e. include/port/{darwin,linux,win32,..}.h). And it seemed
> easier to just have one variable name for all of it.

I am now changing this so that there's an additional 'portname' variable for
this purpose. Otherwise the meson names are used.

Greetings,

Andres Freund

On 24.08.22 17:30, Andres Freund wrote:
>> 258f6dc0a7 Don't hardcode tmp_check/ as test directory for tap tests
>> 8ecc33cf04 Split TESTDIR into TESTLOGDIR and TESTDATADIR
>>
>> I think these patches are split up a bit incorrectly.  If you apply
>> the first patch by itself, then the output appears in tab_comp_dir/
>> directly under the source directory.  And then the second patch moves
>> it to tmp_check/tap_comp_dir/.  If there is an intent to apply these
>> patches separately somehow, this should be cleaned up.
> 
> How is that happening with that version of the patch? The test puts
> tap_comp_dir under TESTDIR, and TESTDIR is $(CURDIR)/tmp_check. There was an
> earlier version of the patch that was split one more time that did have that
> problem, but I don't quite see how that version has it?

Ok, I see now how this works.  It's a bit weird since the meaning of 
TESTDIR is changed.  I'm not sure if this could create cross-branch 
confusion.

>> 97a0b096e8 meson: prereq: Add src/tools/gen_export.pl
>>
>> This produces leading whitespace in the output files that at least on
>> darwin wasn't there before.  See attached patch (0002).  This should
>> be checked again on other platforms as well.
> 
> Hm, to me the indentation as is makes more sense, but ...

Maybe for the 'gnu' format, but on darwin (and others) it's just a flat 
list, so indenting it is pointless.

> I wonder if we should rewrite this in python - I chose perl because I thought
> we could share it, but as you pointed out, that's not possible, because we
> don't want to depend on perl during the autoconf build from a tarball.

Given that the code is already written, I wouldn't do it.  Introducing 
Python into the toolchain would require considerations of minimum 
versions, finding the right binaries, formatting, style checking, etc., 
which would probably be a distraction right now.  I also think that this 
script, whose purpose is to read an input file line by line and print it 
back out slightly differently, is not going to be done better in Python.

Hi,

Attached is v12 of the meson patchset. Lots of improvements:

- initial set of docs for building with meson, contributed by Samay

- PGXS, .pc generation for extensions, making macos tests work when SIP is
  enabled, precompiled headers support are all now separate commits

  as suggested by Peter Eisentraut

- aix, solaris builds work now (both on gcc only)

- most of the operating system specific considerations are now collected in
  one place

  There's still the odd check around, but it's mostly for stuff where it seems
  to make sense to leave decentralized (e.g. do we need to invoke the dtrace
  binary on darwin, using wldap32 on windows, ...)

- split out the existing PG_SYSROOT selection logic from darwin's template
  into src/tools/darwin_sysroot

  Peter E. rightfully complained that the logic I had so far wasn't
  equivalent, and it's finnicky enough that it doesn't seem like a good idea
  to have two copies. Not sure about the location, perhaps it should be in
  config/ instead?

- loads of cleanups, rebasing, etc


The known things that I think need to be fixed before we could consider test
driving this on a larger scale are:

- the various global variables assembled in the toplevel meson.build need
  comments explaining them (e.g. cflags, cflags_sl, ...)

- choice of semaphore API needs to be cleaned up, that should be easy now, but
  I thought that I needed to get a new version out first

- there's a few configure tests denoted with FIXMEs, most importantly I
  haven't caught up to the PGAC_LDAP_SAFE

Greetings,

Andres Freund

Hi,

On 2022-08-27 11:04:47 -0700, Andres Freund wrote:
> - choice of semaphore API needs to be cleaned up, that should be easy now, but
>   I thought that I needed to get a new version out first

Everytime I look at the existing selection code I get confused, which is part
of why I haven't tackled this yet.

If I understand the current logic right, we check for sem_open, sem_init if
and only if PREFERRED_SEMAPHORES is set to UNNAMED_POSIX or NAMED_POSIX (no
template defaults to named).  Which also means that we don't link in rt or
pthread if USE_*_POSIX_SEMAPHORES is set directly in the template, as darwin
does.

I read the configure.ac code combined with the templates as resulting in the
following precedence order:

1) windows uses windows semaphores

2) freebsd, linux use unnamed posix semaphores if available

3) macOS < 10.2 uses named semaphores, without linking in rt/pthread
4) macos >= 10.2 uses sysv semaphores

5) sysv semaphores are used

Correct?


Given that Mac OS 10.2 was released in 2002, I think we can safely consider
that unsupported - even prairiedog was a few years younger... :).  Given the
downsides of named semaphores and that we apparently haven't used that code in
years, I wonder if we should remove it?

However, there has been a thread about defaulting to named semas on openbsd,
but then Tom benchmarked out that that's not going to fly for performance
reasons ([1]).


FWIW, I did notice that netbsd does have working unnamed semaphores. I don't
know how long ago they were added, but they apparently didn't work quite right
in 2018 [1]. No meaningful performance chance in the main regression tests,
I'll run a concurrent check world comparison in the background...


Should the choice be configurable with meson? I'm inclined to say not for now.

Regards,

Andres

[1] https://postgr.es/m/3010886.1634950831%40sss.pgh.pa.us
[2] http://www.polarhome.com/service/man/?qf=sem_init&tf=2&of=NetBSD&sf=3

Hi,

On 2022-08-27 18:02:40 -0700, Andres Freund wrote:
> FWIW, I did notice that netbsd does have working unnamed semaphores. I don't
> know how long ago they were added, but they apparently didn't work quite right
> in 2018 [1]. No meaningful performance chance in the main regression tests,
> I'll run a concurrent check world comparison in the background...

Unnamed ones are substantially worse unfortunately. On an 8 core netbsd 9.3
VM:

sysv:
real    4m39.777s
user    7m35.534s
sys     7m33.831s

unnamed posix
real    5m44.035s
user    7m23.326s
sys     11m58.946s

The difference in system time is even more substantial than the wall clock
time. And repeated runs were even worse.

I also had the ecpg tests hang in one run with unnamed posix semas, until I
killed 'alloc'. Didn't reproduce since though.


So clearly we shouldn't go and start auto-detecting unnamed posix sema
support.

Greetings,

Andres Freund

On Sun, Aug 28, 2022 at 1:39 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-08-27 18:02:40 -0700, Andres Freund wrote:
> > FWIW, I did notice that netbsd does have working unnamed semaphores. I don't
> > know how long ago they were added, but they apparently didn't work quite right
> > in 2018 [1]. No meaningful performance chance in the main regression tests,
> > I'll run a concurrent check world comparison in the background...
>
> Unnamed ones are substantially worse unfortunately. On an 8 core netbsd 9.3
> VM:

I could update my experimental patch to add home made semaphores using
atomics and futexes.  It needs NetBSD 10, though.  Also works on
OpenBSD and macOS.

with_temp_install is repeated twice in prove_check:

> Subject: [PATCH v12 02/15] Split TESTDIR into TESTLOGDIR and TESTDATADIR         
> 
> -   TESTDIR='$(CURDIR)/tmp_check' $(with_temp_install)
>     PGPORT='6$(DEF_PGPORT)' \
> +   TESTLOGDIR='$(CURDIR)/tmp_check/log' $(with_temp_install) \
> +   TESTDATADIR='$(CURDIR)/tmp_check' $(with_temp_install) \
> +   PGPORT='6$(DEF_PGPORT)' \

Before running an individual test like "meson test recovery/017_shm",
it's currently necessary to first manually run "meson test tmp_install".
Is it possible to make that happen automatically ?

You're running tap tests via a python script.  There's no problem with
that, but it's different from what's done by the existing makefiles.
I was able to remove the python indirection - maybe that's better to
talk about on the CI thread?  That moves some setup for TAP tests
(TESTDIR, PATH, cd) from Makefile into the existing perl, which means
less duplication.

Hi,

On 2022-08-28 12:08:07 -0500, Justin Pryzby wrote:
> with_temp_install is repeated twice in prove_check:
>
> > Subject: [PATCH v12 02/15] Split TESTDIR into TESTLOGDIR and TESTDATADIR
> >
> > -   TESTDIR='$(CURDIR)/tmp_check' $(with_temp_install)
> >     PGPORT='6$(DEF_PGPORT)' \
> > +   TESTLOGDIR='$(CURDIR)/tmp_check/log' $(with_temp_install) \
> > +   TESTDATADIR='$(CURDIR)/tmp_check' $(with_temp_install) \
> > +   PGPORT='6$(DEF_PGPORT)' \

Oops, must have screwed up resolving a conflict...


> Before running an individual test like "meson test recovery/017_shm",
> it's currently necessary to first manually run "meson test tmp_install".
> Is it possible to make that happen automatically ?

Not in a trivial way that I found. We don't want to reinstall all the time -
it's *quite* expensive on older machines. We could have a lock file in the
test setup so that the first test run installs it, with the others getting
stalled, but that has pretty obvious disadvantages too (like the test timing
being distorted).

Medium term I think we should consider simply not needing the temp install.

FWIW, if you can do the above as 'meson test tmp_install recovery/017_shm'.


> You're running tap tests via a python script.  There's no problem with
> that, but it's different from what's done by the existing makefiles.
> I was able to remove the python indirection - maybe that's better to
> talk about on the CI thread?  That moves some setup for TAP tests
> (TESTDIR, PATH, cd) from Makefile into the existing perl, which means
> less duplication.

I'm doubtful it's worth removing. You'd need to move removing the files from
the last run into both pg_regress and the tap test infrastructure. And I do
think it's nice to afterwards have markers which tests failed, so we can only
collect their logs.

Greetings,

Andres Freund

I found that the perl test modules are not installed.  See attached 
patch to correct this.

To the patches:

4e15ee0e24 Don't hardcode tmp_check/ as test directory for tap tests
1a3169bc3f Split TESTDIR into TESTLOGDIR and TESTDATADIR

It's a bit weird that the first patch changes the meaning of TESTDIR
and the second patch removes it.  Maybe these patches should be
squashed together?


96d1d0a0cf meson: prereq: Extend gendef.pl in preparation for meson

ok


581721fa99 meson: prereq: Add src/tools/gen_export.pl

Still wondering about the whitespace changes I reported recently, but
that can also be fine-tuned later.


4245cc888e meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build

ok


3afe803e0f meson: prereq: Fix warning compat_informix/rnull.pgc with msvc

ok


ae7733f46c meson: prereq: Move darwin sysroot determination into 
separate file

ok


a1fb97a81b meson: Add meson based buildsystem

I'm not a fan of all this business to protect the two build systems
from each other.  I don't like the build process touching a file under
version control every time.  How necessary is this?  What happens
otherwise?

conversion_helpers.txt: should probably be removed now.

doc/src/sgml/resolv.xsl: I don't understand what this is doing.  Maybe
at least add a comment in the file.

src/common/unicode/meson.build: The comment at the top of the file
should be moved next to the files it is describing (similar to how it
is in the makefile).  I don't see CLDR_VERSION set anywhere.  Is that
part implemented?

src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
(Note that the latter is also used as an input file for text
substitution.  So having another file named *.in next to it would be
super confusing.)

src/tools/find_meson: Could use a brief comment what it does.

src/tools/pgflex: Could use a not-brief comment about what it does,
why it's needed.  Also a comment where it's used.  Also run this
through pycodestyle.

src/tools/rcgen: This is connected with the comment on win32ver.rc.in
above.  We already have this equivalent code in
src/makefiles/Makefile.win32.  Let's figure out a way to share this
code.  (It could be a Perl script, which is already required on
Windows.)  Also pycodestyle.

src/tools/testwrap: also documentation/comments/pycodestyle


cd193eb3e8 meson: ci: Build both with meson and as before

I haven't reviewed this one in detail.  Maybe add a summary in the
commit message, like these are the new jobs, these are the changes to
existing jobs.  It looks like there is more in there than just adding
a few meson jobs.


If the above are addressed, I think this will be just about at the
point where the above patches can be committed.

Everything past these patches I'm mentally postponing right now.

On 24.08.22 17:30, Andres Freund wrote:
>> 0545eec895 meson: Add docs
>>
>> We should think more about how to arrange the documentation.  We
>> probably don't want to copy-and-paste all the introductory and
>> requirements information.  I think we can make this initially much
>> briefer, like the Windows installation chapter.  For example, instead
>> of documenting each setup option again, just mention which ones exist
>> and then point (link) to the configure chapter for details.
> The current docs, including the windows ones, are already hard to follow. I
> think we should take some care to not make the meson bits even more
> confusing. Cross referencing left and right seems problematic from that angle.

If you look at the current structure of the installation chapter

17.1. Short Version
17.2. Requirements
17.3. Getting the Source
17.4. Installation Procedure
17.5. Post-Installation Setup
17.6. Supported Platforms
17.7. Platform-Specific Notes

only 17.1, small parts of 12.2, and 17.4 should differ between make and 
meson.  There is no conceivable reason why the meson installation 
chapter should have a different "Getting the Source" section.  And some 
of the post-installation and platform-specific information doesn't 
appear at all on the meson chapter.

I think we can try to be a bit more ingenious in how we weave this 
together in the best way.  What I really wouldn't want is two separate 
chapters that duplicate the entire process.  I think we could do one 
chapter, like

- Short Version
- Requirements
- Getting the Source
- Installation Procedure
- Installation Procedure using Meson
- Post-Installation Setup
- Supported Platforms
- Platform-Specific Notes

Alternatively, if people prefer two separate chapters, let's think about 
some source-code level techniques to share the common contents.

Hi,

On 2022-08-31 10:28:05 +0200, Peter Eisentraut wrote:
> I found that the perl test modules are not installed.  See attached patch to
> correct this.
>
> To the patches:
>
> 4e15ee0e24 Don't hardcode tmp_check/ as test directory for tap tests
> 1a3169bc3f Split TESTDIR into TESTLOGDIR and TESTDATADIR
>
> It's a bit weird that the first patch changes the meaning of TESTDIR
> and the second patch removes it.  Maybe these patches should be
> squashed together?

Hm, to me they seem topically separate enough, but I don't have a strong
opinion on it.


> 581721fa99 meson: prereq: Add src/tools/gen_export.pl
>
> Still wondering about the whitespace changes I reported recently, but
> that can also be fine-tuned later.

I'll look into it in a bit.


> a1fb97a81b meson: Add meson based buildsystem
>
> I'm not a fan of all this business to protect the two build systems
> from each other.  I don't like the build process touching a file under
> version control every time.  How necessary is this?  What happens
> otherwise?

I added it after just about everyone trying meson hit problems due to
conflicts between (past) in-tree configure builds and meson, due to files left
in tree (picking up the wrong .h files, cannot entirely be fixed with -I
arguments, due to the "" includes). By adding the relevant check to the meson
configure phase, and by triggering meson re-configure whenever an in-tree
configure build is done, these issues can be detected.

It'd of course be nicer to avoid the potential for such conflicts, but that
appears to be a huge chunk of work, see the bison/flex subthread.

So I don't really see an alternative.


> conversion_helpers.txt: should probably be removed now.

Done.


> doc/src/sgml/resolv.xsl: I don't understand what this is doing.  Maybe
> at least add a comment in the file.

It's only used for building epubs. Perhaps I should extract that into a
separate patch as well? The relevant section is:

> #
> # epub
> #
>
> # This was previously implemented using dbtoepub - but that doesn't seem to
> # support running in build != source directory (i.e. VPATH builds already
> # weren't supported).
> if pandoc.found() and xsltproc.found()
>   # XXX: Wasn't able to make pandoc successfully resolve entities
>   # XXX: Perhaps we should just make all targets use this, to avoid repeatedly
>   # building whole thing? It's comparatively fast though.
>   postgres_full_xml = custom_target('postgres-full.xml',
>     input: ['resolv.xsl', 'postgres.sgml'],
>     output: ['postgres-full.xml'],
>     depends: doc_generated + [postgres_sgml_valid],
>     command: [xsltproc, '--path', '@OUTDIR@/', xsltproc_flags,
>               '-o', '@OUTPUT@', '@INPUT@'],
>     build_by_default: false,
>   )

A noted, I couldn't make pandoc resolve our entities, so I used resolv.xsl
them, before calling pandoc.

I'll rename it to resolve-entities.xsl and add a comment.


> src/common/unicode/meson.build: The comment at the top of the file
> should be moved next to the files it is describing (similar to how it
> is in the makefile).

Done.


> I don't see CLDR_VERSION set anywhere.  Is that part implemented?

No, I didn't implement the generation parts of contrib/unaccent. I started
tackling the src/common/unicode bits after John Naylor asked whether that
could be done, but considered that good enough...


> src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
> (Note that the latter is also used as an input file for text
> substitution.  So having another file named *.in next to it would be
> super confusing.)

Yea, this stuff isn't great. I think the better solution, both for meson and
for configure, would be to move to do all the substitution to the C
preprocessor.


> src/tools/find_meson: Could use a brief comment what it does.

Added.


> src/tools/pgflex: Could use a not-brief comment about what it does,
> why it's needed.  Also a comment where it's used.  Also run this
> through pycodestyle.

Working on that.


> cd193eb3e8 meson: ci: Build both with meson and as before
>
> I haven't reviewed this one in detail.  Maybe add a summary in the
> commit message, like these are the new jobs, these are the changes to
> existing jobs.  It looks like there is more in there than just adding
> a few meson jobs.

I don't think we want to commit this as-is. It contains CI for a lot of
platforms - that's very useful for working on meson, but too much for
in-tree. I guess I'll split it into two, one patch for converting a reasonable
subset of the current CI tasks to meson and another to add (back) the current
array of tested platforms.


> If the above are addressed, I think this will be just about at the
> point where the above patches can be committed.

Woo!


> Everything past these patches I'm mentally postponing right now.

Makes sense.

Greetings,

Andres Freund

On Thu, Sep 1, 2022 at 1:12 AM Andres Freund <andres@anarazel.de> wrote:
> [v12]

+# Build a small utility static lib for the parser. This makes it easier to not
+# depend on gram.h already having been generated for most of the other code
+# (which depends on generated headers having been generated). The generation
+# of the parser is slow...

It's not obvious whether this is intended to be a Meson-only
optimization or a workaround for something awkward to specify.

+  # FIXME: -output option is only available in perl 5.9.3 - but that's
+  # probably a fine minimum requirement?

Since we've retired some buildfarm animals recently, it seems the
oldest perl there is 5.14? ... which came out in 2011, so it seems
later on we could just set that as the minimum.

-- 
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-09-02 14:17:26 +0700, John Naylor wrote:
> On Thu, Sep 1, 2022 at 1:12 AM Andres Freund <andres@anarazel.de> wrote:
> > [v12]
> 
> +# Build a small utility static lib for the parser. This makes it easier to not
> +# depend on gram.h already having been generated for most of the other code
> +# (which depends on generated headers having been generated). The generation
> +# of the parser is slow...
> 
> It's not obvious whether this is intended to be a Meson-only
> optimization or a workaround for something awkward to specify.

It is an optimization. The parser generation is by far the slowest part of a
build. If other files can only be compiled once gram.h is generated, there's a
long initial period where little can happen. So instead of having all .c files
have a dependency on gram.h having been generated, the above makes only
scan.c, gram.c compilation depend on gram.h.  It only matters for the first
compilation, because such dependencies are added as order-only dependencies,
supplanted by more precise compiler generated dependencies after.

See the attached dep and nodep.png. That's ui.perfetto.dev displaying the
.ninja_log file, showing the time for building the backend on my
workstation. The difference is probably apparent.

It's still pretty annoying that so much of the build is initially idle,
waiting for genbki.pl to finish.

Part of that is due to some ugly dependencies of src/common on backend headers
that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
catalog/pg_tablespace_d.h). Looks like it'd not be hard to get at least the
_shlib version of src/common and libpq build without waiting for that. But for
all the backend code I don't really see a way, so it'd be nice to make genbki
faster at some point.


> +  # FIXME: -output option is only available in perl 5.9.3 - but that's
> +  # probably a fine minimum requirement?
> 
> Since we've retired some buildfarm animals recently, it seems the
> oldest perl there is 5.14? ... which came out in 2011, so it seems
> later on we could just set that as the minimum.

At the moment we document 5.8.3 as our minimum, supposedly based on some
buildfarm animal - but that's probably outdated. Perhaps time to start that
discussion? Or maybe it's fine to just have the meson stuff have that
dependency for now. Seems exceedingly unlikely anybody would care.

Greetings,

Andres Freund

Hi,

On 2022-09-02 09:35:15 -0700, Andres Freund wrote:
> Part of that is due to some ugly dependencies of src/common on backend headers
> that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
> catalog/pg_tablespace_d.h). Looks like it'd not be hard to get at least the
> _shlib version of src/common and libpq build without waiting for that. But for
> all the backend code I don't really see a way, so it'd be nice to make genbki
> faster at some point.

This reminded me of a question I had. Requires a bit of an intro...


Because ninja's build specification ends up as a fairly clean DAG, and because
it also collects compiler generated dependencies as a DAG, it's possible to
check whether the build specification contains sufficient dependencies.

Before ninja 1.11 there was
https://github.com/llvm/llvm-project/blob/main/llvm/utils/check_ninja_deps.py
and ninja 1.11 has "ninja -t missingdeps" built in.

Intentionally removing some of the dependencies to show the output:

$ ninja -t missingdeps
Missing dep: src/interfaces/ecpg/preproc/ecpg.p/.._ecpglib_typename.c.o uses src/include/catalog/pg_type_d.h (generated
byCUSTOM_COMMAND)
 
...
Missing dep: src/bin/scripts/reindexdb.p/reindexdb.c.o uses src/include/catalog/pg_class_d.h (generated by
CUSTOM_COMMAND)
Missing dep: contrib/oid2name/oid2name.p/oid2name.c.o uses src/include/catalog/pg_class_d.h (generated by
CUSTOM_COMMAND)
Missing dep: contrib/vacuumlo/vacuumlo.p/vacuumlo.c.o uses src/include/catalog/pg_class_d.h (generated by
CUSTOM_COMMAND)
Missing dep: src/test/modules/libpq_pipeline/libpq_pipeline.p/libpq_pipeline.c.o uses src/include/catalog/pg_type_d.h
(generatedby CUSTOM_COMMAND)
 
Processed 2299 nodes.
Error: There are 62 missing dependency paths.
62 targets had depfile dependencies on 25 distinct generated inputs (from 1 rules)  without a non-depfile dep path to
thegenerator.
 
There might be build flakiness if any of the targets listed above are built alone, or not late enough, in a clean
outputdirectory.
 

Obviously that can only work after building, as the compiler generated
dependencies are needed.

I find this exceedingly helpful, because it supplies a very high guarantee
that the build specification will not fail on a different machine due to
different performance characteristics.

The question:

Is it worth running ninja -t missingdeps as a test? At the time we run tests
we'll obviously have built and thus collected "real" dependencies, so we would
have the necessary information to determine whether dependencies are missing.
I think it'd be fine to do so only for ninja >= 1.11, rather than falling back
to the llvm python implementation, which is much slower (0.068s vs
3.760s). And also because it's not as obvious how to include the python script.

Alternatively, we could just document that ninja -t missingdeps is worth
running. Perhaps at the top of the toplevel build.meson file?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-09-02 14:17:26 +0700, John Naylor wrote:
>> +  # FIXME: -output option is only available in perl 5.9.3 - but that's
>> +  # probably a fine minimum requirement?
>> 
>> Since we've retired some buildfarm animals recently, it seems the
>> oldest perl there is 5.14? ... which came out in 2011, so it seems
>> later on we could just set that as the minimum.

> At the moment we document 5.8.3 as our minimum, supposedly based on some
> buildfarm animal - but that's probably outdated.

Yeah, definitely.  prairiedog was the only animal running such an old
version, and it's gone.  I don't think we have anything testing ancient
bison or flex anymore, either.  I'm a fan of actually testing whatever
we claim as the minimum supported version of any tool, so there's some
work to be done here, on buildfarm config or docs or both.

            regards, tom lane

Hi,

Split off from the meson thread at https://postgr.es/m/990067.1662138678%40sss.pgh.pa.us

On 2022-09-02 13:11:18 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-09-02 14:17:26 +0700, John Naylor wrote:
> >> +  # FIXME: -output option is only available in perl 5.9.3 - but that's
> >> +  # probably a fine minimum requirement?
> >>
> >> Since we've retired some buildfarm animals recently, it seems the
> >> oldest perl there is 5.14? ... which came out in 2011, so it seems
> >> later on we could just set that as the minimum.
>
> > At the moment we document 5.8.3 as our minimum, supposedly based on some
> > buildfarm animal - but that's probably outdated.
>
> Yeah, definitely.  prairiedog was the only animal running such an old
> version, and it's gone.  I don't think we have anything testing ancient
> bison or flex anymore, either.  I'm a fan of actually testing whatever
> we claim as the minimum supported version of any tool, so there's some
> work to be done here, on buildfarm config or docs or both.

5.8.3 is from 2004-Jan-14, that's impressive :). I don't see any benefit in
setting up a buildfarm animal running that old a version.

For the meson stuff it'd suffice to set 5.9.3. as the minimum version for
plperl (or I could try to work around it). However, supporting a perl version
from 2006-Jan-28 doesn't strike me as particularly useful either.


Relevent somewhat recent discussion / work:
https://postgr.es/m/87y278s6iq.fsf%40wibble.ilmari.org
https://www.postgresql.org/message-id/E1mYY6Z-0006OL-QN%40gemulon.postgresql.org


I looked at which buildfarm animals currently use 5.14 (mentioned by John),
and it's frogfish, snapper and skate. The latter two do build with plperl.


I started a query on the buildfarm machine to collect the perl versions, but
it's just awfully slow...

Greetings,

Andres Freund

Hi John,

Are you planning to press ahead with these?


> Subject: [PATCH v4 01/11] Preparatory refactoring for compiling guc-file.c
>  standalone
> Subject: [PATCH v4 02/11] Move private declarations shared between guc.c and
>  guc-file.l to new header
> Subject: [PATCH v4 03/11] Build guc-file.c standalone

01-03 are a bit more complicated, but still look not far off. There's a FIXME
about failing headercheck.


> Subject: [PATCH v4 04/11] Build bootscanner.c standalone
> Subject: [PATCH v4 05/11] Build repl_scanner.c standalone
> Subject: [PATCH v4 06/11] Build syncrep_scanner.c standalone
> Subject: [PATCH v4 07/11] Build specscanner.c standalone
> Subject: [PATCH v4 08/11] Build exprscan.c standalone

LGTM


> Subject: [PATCH v4 09/11] Build cubescan.c standalone
> 
> Pass scanbuflen as a parameter to yyparse rather than
> resorting to a global variable.

Nice.


> Subject: [PATCH v4 10/11] Build segscan.c standalone
> Subject: [PATCH v4 11/11] Build jsonpath_scan.c standalone

LGTM.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I started a query on the buildfarm machine to collect the perl versions, but
> it's just awfully slow...

This is from March, but it's probably still accurate enough.

            regards, tom lane

    sysname    |      snapshot       |                      l
---------------+---------------------+----------------------------------------------
 alabio        | 2022-03-24 20:00:07 | configure: using perl 5.28.1
 anole         | 2022-03-24 10:40:08 | configure: using perl 5.8.8
 avocet        | 2022-03-25 01:15:48 | configure: using perl 5.26.1
 ayu           | 2022-03-24 22:40:55 | configure: using perl 5.24.1
 barbastella   | 2022-03-25 00:26:48 | configure: using perl 5.26.1
 barbthroat    | 2022-03-24 12:24:39 | configure: using perl 5.26.1
 batfish       | 2022-03-24 19:45:25 | configure: using perl 5.22.1
 bichir        | 2022-03-07 15:30:09 | configure: using perl 5.26.1
 bonito        | 2022-03-24 23:29:26 | configure: using perl 5.28.2
 boomslang     | 2022-03-24 09:14:08 | configure: using perl 5.32.1
 bufflehead    | 2022-02-16 19:45:26 | configure: using perl 5.18.2
 buri          | 2022-03-25 00:34:13 | configure: using perl 5.16.3
 butterflyfish | 2022-03-25 01:00:16 | configure: using perl 5.24.1
 caiman        | 2022-03-24 23:24:08 | configure: using perl 5.34.1
 calliphoridae | 2022-03-25 00:55:17 | configure: using perl 5.34.0
 cardinalfish  | 2022-03-24 02:10:06 | configure: using perl 5.34.0
 cavefish      | 2022-03-24 04:29:49 | configure: using perl 5.26.1
 chimaera      | 2022-03-24 02:14:19 | configure: using perl 5.24.1
 chipmunk      | 2022-03-20 21:57:01 | configure: using perl 5.20.2
 chub          | 2022-03-24 16:10:02 | configure: using perl 5.16.3
 ciconia       | 2022-02-03 11:45:11 | configure: using perl 5.26.3
 clam          | 2022-03-24 22:27:35 | configure: using perl 5.16.3
 conchuela     | 2022-03-24 23:35:43 | configure: using perl 5.32.1
 copperhead    | 2022-03-23 20:49:03 | configure: using perl 5.32.1
 crake         | 2022-03-25 00:32:20 | Mar 24 20:32:22 configure: using perl 5.34.0
 culicidae     | 2022-03-25 00:55:39 | configure: using perl 5.34.0
 cuon          | 2022-03-24 07:41:28 | configure: using perl 5.22.1
 curculio      | 2022-03-25 01:15:30 | configure: using perl 5.20.2
 dangomushi    | 2022-03-24 19:43:23 | configure: using perl 5.34.0
 demoiselle    | 2022-03-24 15:13:03 | configure: using perl 5.26.1
 desmoxytes    | 2022-03-25 00:56:03 | configure: using perl 5.34.0
 devario       | 2022-03-24 13:47:09 | configure: using perl 5.34.0
 dhole         | 2022-03-24 05:57:52 | configure: using perl 5.16.3
 dragonet      | 2022-03-24 19:34:05 | configure: using perl 5.34.0
 eelpout       | 2022-03-25 00:22:33 | configure: using perl 5.32.1
 elasmobranch  | 2022-03-24 03:10:07 | configure: using perl 5.26.1
 elver         | 2022-02-01 22:06:10 | configure: using perl 5.32.1
 fairywren     | 2022-03-22 21:23:18 | configure: using perl 5.24.3
 flaviventris  | 2022-03-25 01:24:08 | configure: using perl 5.34.0
 florican      | 2022-03-25 00:29:01 | configure: using perl 5.32.1
 francolin     | 2022-03-25 00:55:12 | configure: using perl 5.34.0
 frogfish      | 2022-02-26 17:38:06 | configure: using perl 5.14.2
 gadwall       | 2022-02-16 11:50:06 | configure: using perl 5.26.1
 gaur          | 2022-03-19 15:00:54 | configure: using perl 5.8.9
 gharial       | 2022-03-24 18:32:22 | configure: using perl 5.8.8
 gombessa      | 2022-03-15 02:58:20 | configure: using perl 5.30.3
 grison        | 2022-03-24 15:12:14 | configure: using perl 5.24.1
 guaibasaurus  | 2022-03-25 00:20:05 | configure: using perl 5.28.1
 gull          | 2022-03-19 08:24:23 | configure: using perl 5.32.1
 haddock       | 2022-02-16 23:36:11 | configure: using perl 5.22.4
 hake          | 2022-02-24 02:33:48 | configure: using perl 5.24.3
 halibut       | 2022-03-15 04:04:28 | configure: using perl 5.30.1
 handfish      | 2022-03-24 23:46:53 | configure: using perl 5.34.0
 hippopotamus  | 2022-03-25 00:25:02 | configure: using perl 5.26.1
 hornet        | 2022-03-21 06:06:32 | configure: using perl 5.28.1
 hoverfly      | 2022-03-21 22:39:52 | configure: using perl 5.28.1
 ibisbill      | 2022-03-24 08:15:35 | configure: using perl 5.28.1
 idiacanthus   | 2022-03-25 00:56:04 | configure: using perl 5.34.0
 jabiru        | 2022-03-25 00:25:27 | configure: using perl 5.32.1
 jacana        | 2022-03-22 02:53:11 | configure: using perl 5.26.3
 jay           | 2022-03-24 21:59:03 | configure: using perl 5.26.1
 kittiwake     | 2022-03-24 17:21:42 | configure: using perl 5.28.1
 komodoensis   | 2022-03-24 13:08:04 | configure: using perl 5.34.0
 lapwing       | 2022-03-24 23:34:48 | configure: using perl 5.14.2
 loach         | 2022-03-24 22:17:28 | configure: using perl 5.32.1
 locust        | 2022-03-19 22:17:53 | configure: using perl 5.8.8
 longfin       | 2022-03-25 00:59:30 | configure: using perl 5.30.2
 lorikeet      | 2022-03-24 23:13:02 | configure: using perl 5.32.1
 mandrill      | 2022-03-21 13:37:42 | configure: using perl 5.28.1
 mantid        | 2022-03-25 01:07:08 | configure: using perl 5.16.3
 marabou       | 2022-02-18 12:00:08 | configure: using perl 5.32.1
 massasauga    | 2022-03-25 00:25:21 | configure: using perl 5.30.2
 mereswine     | 2022-03-20 08:38:52 | configure: using perl 5.32.1
 moonjelly     | 2022-03-25 01:23:36 | configure: using perl 5.30.0
 morepork      | 2022-03-24 21:45:27 | configure: using perl 5.32.1
 mule          | 2022-03-25 00:30:04 | configure: using perl 5.32.1
 mussurana     | 2022-03-24 08:15:46 | configure: using perl 5.24.1
 mylodon       | 2022-03-25 00:55:33 | configure: using perl 5.34.0
 myna          | 2022-03-25 01:00:17 | configure: using perl 5.28.0
 peripatus     | 2022-03-25 01:20:13 | configure: using perl 5.32.1
 petalura      | 2022-03-24 13:20:03 | configure: using perl 5.34.0
 phycodurus    | 2022-03-24 20:30:05 | configure: using perl 5.34.0
 piculet       | 2022-03-25 00:56:27 | configure: using perl 5.34.0
 pintail       | 2022-02-16 19:22:59 | configure: using perl 5.26.1
 pipistrelles  | 2022-03-24 18:23:50 | configure: using perl 5.26.1
 plover        | 2022-03-15 03:15:39 | configure: using perl 5.32.1
 pogona        | 2022-03-24 18:10:04 | configure: using perl 5.34.0
 pollock       | 2022-03-24 04:37:29 | configure: using perl 5.34.0
 prairiedog    | 2022-03-23 05:04:02 | configure: using perl 5.8.3
 prion         | 2022-03-25 00:23:14 | configure: using perl 5.16.3
 queensnake    | 2022-03-24 23:40:25 | configure: using perl 5.26.3
 quokka        | 2022-03-24 21:52:28 | configure: using perl 5.16.3
 rhinoceros    | 2022-03-25 00:52:15 | configure: using perl 5.16.3
 rorqual       | 2022-03-25 00:55:35 | configure: using perl 5.34.0
 scoter        | 2022-03-24 19:08:00 | configure: using perl 5.16.3
 seawasp       | 2022-03-25 00:17:48 | configure: using perl 5.30.0
 serinus       | 2022-03-24 19:00:07 | configure: using perl 5.34.0
 sheartail     | 2022-03-24 08:49:14 | configure: using perl 5.26.3
 shelduck      | 2022-03-11 07:01:22 | configure: using perl 5.18.2
 sidewinder    | 2022-03-25 01:05:30 | configure: using perl 5.34.0
 sifaka        | 2022-03-25 01:26:54 | configure: using perl 5.30.3
 skate         | 2022-03-24 08:12:31 | configure: using perl 5.14.2
 skink         | 2022-03-24 09:39:42 | configure: using perl 5.34.0
 snakefly      | 2022-03-25 00:56:04 | configure: using perl 5.16.3
 snapper       | 2022-03-24 17:53:00 | configure: using perl 5.14.2
 spurfowl      | 2022-03-25 00:57:10 | configure: using perl 5.22.1
 steamerduck   | 2022-03-24 06:25:03 | configure: using perl 5.26.1
 sungazer      | 2022-03-19 21:46:09 | configure: using perl 5.28.1
 tadarida      | 2022-03-24 17:38:28 | configure: using perl 5.24.1
 takin         | 2022-02-16 08:11:53 | configure: using perl 5.18.2
 tern          | 2022-03-21 16:16:31 | configure: using perl 5.28.1
 thorntail     | 2022-03-24 23:51:01 | configure: using perl 5.34.0
 topminnow     | 2022-03-22 20:01:49 | configure: using perl 5.20.2
 trilobite     | 2022-03-25 00:47:19 | configure: using perl 5.26.1
 urocryon      | 2022-03-24 06:32:49 | configure: using perl 5.24.1
 vulpes        | 2022-03-24 09:37:24 | configure: using perl 5.26.2
 warbler       | 2022-03-15 08:26:07 | configure: using perl 5.32.1
 wobbegong     | 2022-03-24 21:38:58 | configure: using perl 5.26.2
 woodstar      | 2022-03-24 20:55:28 | configure: using perl 5.26.3
 wrasse        | 2022-03-25 00:17:27 | configure: using perl 5.30.1
 xenodermus    | 2022-03-24 22:00:05 | configure: using perl 5.34.0
(121 rows)

Hi,

On 2022-09-02 14:31:57 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I started a query on the buildfarm machine to collect the perl versions, but
> > it's just awfully slow...
>
> This is from March, but it's probably still accurate enough.

Thanks.

Mine did just finish. Over the last month there were the following perl
version on HEAD:

 perl_version |     last_report     |
                                        array_agg
 

--------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 {5,8,3}      | 2022-08-04 09:38:04 | {prairiedog}
 {5,14,2}     | 2022-09-02 16:40:12 | {skate,lapwing,snapper,frogfish}
 {5,16,3}     | 2022-09-02 16:52:17 | {prion,dhole,buri,parula,mantid,chub,clam,snakefly,rhinoceros,quokka}
 {5,18,2}     | 2022-09-02 06:42:13 | {shelduck}
 {5,20,2}     | 2022-09-02 16:15:34 | {curculio,chipmunk,topminnow}
 {5,22,1}     | 2022-09-02 16:02:11 | {spurfowl,cuon,batfish}
 {5,24,1}     | 2022-09-02 17:00:17 | {urocryon,grison,mussurana,butterflyfish,ayu,chimaera,tadarida}
 {5,24,3}     | 2022-09-02 09:04:12 | {fairywren}
 {5,26,1}     | 2022-09-02 18:40:18 |
{elasmobranch,avocet,bichir,blossomcrown,trilobite,cavefish,cotinga,demoiselle,perch,hippopotamus,jay}
 {5,26,2}     | 2022-09-02 09:02:03 | {vulpes,wobbegong}
 {5,26,3}     | 2022-09-02 12:04:01 | {jacana}
 {5,28,0}     | 2022-09-02 17:00:17 | {myna}
 {5,28,1}     | 2022-09-02 16:02:01 | {sungazer,hornet,hoverfly,ibisbill,kittiwake,mandrill,tern}
 {5,28,2}     | 2022-09-01 23:39:33 | {bonito}
 {5,30,0}     | 2022-09-02 14:16:16 | {branta,moonjelly,urutau,seawasp}
 {5,30,1}     | 2022-09-02 02:59:06 | {wrasse}
 {5,30,2}     | 2022-09-02 16:05:24 | {massasauga}
 {5,30,3}     | 2022-09-02 17:00:06 | {longfin,sifaka,gombessa}
 {5,32,0}     | 2022-09-02 16:00:05 | {margay}
 {5,32,1}     | 2022-09-02 17:49:36 |
{lorikeet,alabio,guaibasaurus,eelpout,tayra,peripatus,plover,gull,mereswine,warbler,morepork,mule,loach,boomslang,florican,copperhead,conchuela}
 {5,34,0}     | 2022-09-02 16:30:04 |
{culicidae,komodoensis,grassquit,mamba,francolin,mylodon,olingo,flaviventris,petalura,phycodurus,piculet,pogona,dragonet,devario,desmoxytes,rorqual,serinus,kestrel,crake,skink,chickadee,cardinalfish,tamandua,xenodermus,thorntail,calliphoridae,idiacanthus}
 {5,34,1}     | 2022-09-02 16:05:33 | {sidewinder,malleefowl,pollock}
 {5,36,0}     | 2022-09-02 03:01:08 | {dangomushi,caiman}
(23 rows)

5.14 would be a trivial lift as far as the buildfarm is concerned. The Debian
7 animals couldn't trivially be updated to a newer perl. It's from 2013-05-04,
so I wouldn't feel bad about dropping support for it - but probably wouldn't
personally bother just for this.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> 5.14 would be a trivial lift as far as the buildfarm is concerned.

Yeah, that seems like a reasonable new minimum for Perl.  I might
see about setting up an animal running 5.14.0, just so we can say
"5.14" in the docs without fine print.

            regards, tom lane

> On 2 Sep 2022, at 21:11, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Andres Freund <andres@anarazel.de> writes:
>> 5.14 would be a trivial lift as far as the buildfarm is concerned.
> 
> Yeah, that seems like a reasonable new minimum for Perl.  I might
> see about setting up an animal running 5.14.0, just so we can say
> "5.14" in the docs without fine print.

Maybe perlbrew can be used, as per the instructions in src/test/perl/README?

--
Daniel Gustafsson        https://vmware.com/

On Sat, Sep 3, 2022 at 1:29 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi John,
>
> Are you planning to press ahead with these?

I was waiting for feedback on the latest set, so tomorrow I'll see
about the FIXME and remove the leftover bogus include. I was thinking
of applying the guc-file patches separately and then squashing the
rest since they're *mostly* mechanical:

> > Subject: [PATCH v4 01/11] Preparatory refactoring for compiling guc-file.c
> >  standalone
> > Subject: [PATCH v4 02/11] Move private declarations shared between guc.c and
> >  guc-file.l to new header
> > Subject: [PATCH v4 03/11] Build guc-file.c standalone
>
> 01-03 are a bit more complicated, but still look not far off. There's a FIXME
> about failing headercheck.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On Sat, Sep 3, 2022 at 1:29 AM Andres Freund <andres@anarazel.de> wrote:

> > Subject: [PATCH v4 01/11] Preparatory refactoring for compiling guc-file.c
> >  standalone
> > Subject: [PATCH v4 02/11] Move private declarations shared between guc.c and
> >  guc-file.l to new header
> > Subject: [PATCH v4 03/11] Build guc-file.c standalone
>
> 01-03 are a bit more complicated, but still look not far off. There's a FIXME
> about failing headercheck.

Fixed by adding utils/guc.h to the new internal header, which now
lives in the same directory as guc.c and guc-file.l, similar to how I
did json path in the last patch. Also removed the bogus include from
v4 to . Pushed 01 and 02 separately, then squashed and pushed the
rest.


--
John Naylor
EDB: http://www.enterprisedb.com

On Fri, Sep 2, 2022 at 11:35 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-09-02 14:17:26 +0700, John Naylor wrote:
> > On Thu, Sep 1, 2022 at 1:12 AM Andres Freund <andres@anarazel.de> wrote:
> > > [v12]
> >
> > +# Build a small utility static lib for the parser. This makes it easier to not
> > +# depend on gram.h already having been generated for most of the other code
> > +# (which depends on generated headers having been generated). The generation
> > +# of the parser is slow...
> >
> > It's not obvious whether this is intended to be a Meson-only
> > optimization or a workaround for something awkward to specify.
>
> It is an optimization. The parser generation is by far the slowest part of a
> build. If other files can only be compiled once gram.h is generated, there's a
> long initial period where little can happen. So instead of having all .c files
> have a dependency on gram.h having been generated, the above makes only
> scan.c, gram.c compilation depend on gram.h.  It only matters for the first
> compilation, because such dependencies are added as order-only dependencies,
> supplanted by more precise compiler generated dependencies after.

Okay, I think the comment could include some of this info for clarity.

> It's still pretty annoying that so much of the build is initially idle,
> waiting for genbki.pl to finish.
>
> Part of that is due to some ugly dependencies of src/common on backend headers
> that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
> catalog/pg_tablespace_d.h).

Technically, *_d.h headers are not backend, that's why it's safe to
include them anywhere. relpath.c in its current form has to know the
tablespace OIDs, which I guess is what you think is ugly. (I agree
it's not great)

> Looks like it'd not be hard to get at least the
> _shlib version of src/common and libpq build without waiting for that. But for
> all the backend code I don't really see a way, so it'd be nice to make genbki
> faster at some point.

The attached gets me a ~15% reduction in clock time by having
Catalog.pm parse the .dat files in one sweep, when we don't care about
formatting, i.e. most of the time:

master:
User time (seconds): 0.48
Maximum resident set size (kbytes): 36112

patch:
User time (seconds): 0.41
Maximum resident set size (kbytes): 35808

That's pretty simple -- I think going beyond that would require some
perl profiling.

-- 
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-09-04 12:16:10 +0700, John Naylor wrote:
> Pushed 01 and 02 separately, then squashed and pushed the rest.

Thanks a lot! It does look a good bit cleaner to me now.

I think, as a followup improvement, we should move gramparse.h to
src/backend/parser, and stop installing gram.h, gramparse.h. gramparse.h
already had this note:

 * NOTE: this file is only meant to be included in the core parsing files,
 * i.e., parser.c, gram.y, and scan.l.
 * Definitions that are needed outside the core parser should be in parser.h.

What do you think?


I looked for projects including gramparse.h ([1], and found libpg-query, pgpool,
slony1 and oracfe:
- libpg-query, pgpool are partial copies of our code so will catch up when
  they sync up,
- slony1's [2] is a configure check, one that long seems outdated, because it's
  grepping for standard_conforming strings, which was moved out in 6566e37e027
  in 2009.
- As far as I can tell oracfe's include in sqlscan.l is vistigial, it compiles
  without it. And the include in parse_keywords.c is just required because it
  needs to include parser/scanner.h.

Greetings,

Andres Freund

[1] https://codesearch.debian.net/search?q=gramparse.h&literal=1&perpkg=1
[2]
https://git.postgresql.org/gitweb/?p=slony1-engine.git;a=blob;f=config/acx_libpq.m4;h=7653357c0a731e36ec637df5ab378832d9279c19;hb=HEAD#l530

Hi,

On 2022-09-04 13:12:52 +0700, John Naylor wrote:
> On Fri, Sep 2, 2022 at 11:35 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2022-09-02 14:17:26 +0700, John Naylor wrote:
> > > On Thu, Sep 1, 2022 at 1:12 AM Andres Freund <andres@anarazel.de> wrote:
> > > > [v12]
> > >
> > > +# Build a small utility static lib for the parser. This makes it easier to not
> > > +# depend on gram.h already having been generated for most of the other code
> > > +# (which depends on generated headers having been generated). The generation
> > > +# of the parser is slow...
> > >
> > > It's not obvious whether this is intended to be a Meson-only
> > > optimization or a workaround for something awkward to specify.
> >
> > It is an optimization. The parser generation is by far the slowest part of a
> > build. If other files can only be compiled once gram.h is generated, there's a
> > long initial period where little can happen. So instead of having all .c files
> > have a dependency on gram.h having been generated, the above makes only
> > scan.c, gram.c compilation depend on gram.h.  It only matters for the first
> > compilation, because such dependencies are added as order-only dependencies,
> > supplanted by more precise compiler generated dependencies after.
> 
> Okay, I think the comment could include some of this info for clarity.

Working on that.


> > It's still pretty annoying that so much of the build is initially idle,
> > waiting for genbki.pl to finish.
> >
> > Part of that is due to some ugly dependencies of src/common on backend headers
> > that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
> > catalog/pg_tablespace_d.h).
> 
> Technically, *_d.h headers are not backend, that's why it's safe to
> include them anywhere. relpath.c in its current form has to know the
> tablespace OIDs, which I guess is what you think is ugly. (I agree
> it's not great)

Yea, I'm not saying it's unsafe in a produces-wrong-results way, just that it
seems architecturally dubious / circular.


> > Looks like it'd not be hard to get at least the
> > _shlib version of src/common and libpq build without waiting for that. But for
> > all the backend code I don't really see a way, so it'd be nice to make genbki
> > faster at some point.
> 
> The attached gets me a ~15% reduction in clock time by having
> Catalog.pm parse the .dat files in one sweep, when we don't care about
> formatting, i.e. most of the time:

Cool. Seems worthwhile.

Greetings,

Andres Freund

On Mon, Sep 5, 2022 at 4:11 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-09-04 13:12:52 +0700, John Naylor wrote:
> > On Fri, Sep 2, 2022 at 11:35 PM Andres Freund <andres@anarazel.de> wrote:
> > >
> > > Hi,
> > >
> > > On 2022-09-02 14:17:26 +0700, John Naylor wrote:
> > > > On Thu, Sep 1, 2022 at 1:12 AM Andres Freund <andres@anarazel.de> wrote:
> > > > > [v12]
> > > >
> > > > +# Build a small utility static lib for the parser. This makes it easier to not
> > > > +# depend on gram.h already having been generated for most of the other code
> > > > +# (which depends on generated headers having been generated). The generation
> > > > +# of the parser is slow...
> > > >
> > > > It's not obvious whether this is intended to be a Meson-only
> > > > optimization or a workaround for something awkward to specify.
> > >
> > > It is an optimization. The parser generation is by far the slowest part of a
> > > build. If other files can only be compiled once gram.h is generated, there's a
> > > long initial period where little can happen. So instead of having all .c files
> > > have a dependency on gram.h having been generated, the above makes only
> > > scan.c, gram.c compilation depend on gram.h.  It only matters for the first
> > > compilation, because such dependencies are added as order-only dependencies,
> > > supplanted by more precise compiler generated dependencies after.
> >
> > Okay, I think the comment could include some of this info for clarity.
>
> Working on that.
>
>
> > > It's still pretty annoying that so much of the build is initially idle,
> > > waiting for genbki.pl to finish.
> > >
> > > Part of that is due to some ugly dependencies of src/common on backend headers
> > > that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
> > > catalog/pg_tablespace_d.h).
> >
> > Technically, *_d.h headers are not backend, that's why it's safe to
> > include them anywhere. relpath.c in its current form has to know the
> > tablespace OIDs, which I guess is what you think is ugly. (I agree
> > it's not great)
>
> Yea, I'm not saying it's unsafe in a produces-wrong-results way, just that it
> seems architecturally dubious / circular.
>
>
> > > Looks like it'd not be hard to get at least the
> > > _shlib version of src/common and libpq build without waiting for that. But for
> > > all the backend code I don't really see a way, so it'd be nice to make genbki
> > > faster at some point.
> >
> > The attached gets me a ~15% reduction in clock time by having
> > Catalog.pm parse the .dat files in one sweep, when we don't care about
> > formatting, i.e. most of the time:
>
> Cool. Seems worthwhile.

Okay, here's a cleaned up version with more idiomatic style and a new
copy of the perlcritic exception.

Note that the indentation hasn't changed. My thought there: perltidy
will be run again next year, at which time it will be part of a listed
whitespace-only commit. Any objections, since that could confuse
someone before then?

-- 
John Naylor
EDB: http://www.enterprisedb.com

On Mon, Sep 5, 2022 at 1:18 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-09-04 12:16:10 +0700, John Naylor wrote:
> > Pushed 01 and 02 separately, then squashed and pushed the rest.
>
> Thanks a lot! It does look a good bit cleaner to me now.
>
> I think, as a followup improvement, we should move gramparse.h to
> src/backend/parser, and stop installing gram.h, gramparse.h. gramparse.h
> already had this note:
>
>  * NOTE: this file is only meant to be included in the core parsing files,
>  * i.e., parser.c, gram.y, and scan.l.
>  * Definitions that are needed outside the core parser should be in parser.h.
>
> What do you think?

+1 for the concept, but haven't looked at the details.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On 02.09.22 01:12, samay sharma wrote:
> So, I was thinking of the following structure:
> - Supported Platforms
> - Getting the Source
> - Building with make and autoconf
>    -- Short version
>    -- Requirements
>    -- Installation Procedure and it's subsections
> - Building with Meson
>    -- Short version
>    -- Requirements
>    -- Installation Procedure and it's subsections
> - Post-installation Setup
> - Platform specific notes

I like that.

> As a follow up patch, we could also try to fit the Windows part into 
> this model. We could add a Building with visual C++ or Microsoft windows 
> SDK section. It doesn't have a short version but follows the remaining 
> template of requirements and installation procedure subsections 
> (Building, Cleaning and Installing and Running Regression tests) well.

We were thinking about removing the old Windows build system for PG 16. 
Let's see how that goes.  Otherwise, yes, that would be good as well.

On 02.09.22 19:16, samay sharma wrote:
> Another thing I'd like input on. A common question I've heard from 
> people who've tried out the docs is How do we do the equivalent of X in 
> make with meson. As meson will be new for a bunch of people who are 
> fluent with make, I won't be surprised if this is a common ask. To 
> address that, I was planning to add a page to specify the key things one 
> needs to keep in mind while "migrating" from make to meson and having a 
> translation table of commonly used commands.
> 
> I was planning to add it in the meson section, but if we go ahead with 
> the structure proposed above, it doesn't fit it into one as cleanly. 
> Maybe, it still goes in the meson section? Thoughts?

This could go into the wiki.

For example, we have 
<https://wiki.postgresql.org/wiki/Working_with_Git>, which was added 
during the CVS->Git transition.

This avoids that we make the PostgreSQL documentation a substitute 
manual for a third-party product.

On 31.08.22 20:11, Andres Freund wrote:
>> src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
>> (Note that the latter is also used as an input file for text
>> substitution.  So having another file named *.in next to it would be
>> super confusing.)
> Yea, this stuff isn't great. I think the better solution, both for meson and
> for configure, would be to move to do all the substitution to the C
> preprocessor.

Yeah, I think if we can get rid of the evil date-based versioning, then
this could be done like

diff --git a/src/makefiles/Makefile.win32 b/src/makefiles/Makefile.win32
index 17d6819644..609156382f 100644
--- a/src/makefiles/Makefile.win32
+++ b/src/makefiles/Makefile.win32
@@ -65,21 +65,12 @@ endif
  # win32ver.rc or furnish a rule for generating one.  Set $(PGFILEDESC) to
  # signal win32ver.rc availability to the dll build rule below.
  ifndef PGXS
-win32ver.rc: $(top_srcdir)/src/port/win32ver.rc
-   sed -e 's;FILEDESC;$(PGFILEDESC);' \
-       -e 's;VFT_APP;$(PGFTYPE);' \
-       -e 's;_ICO_;$(PGICOSTR);' \
-       -e 's;\(VERSION.*\),0 *$$;\1,'`date '+%y%j' | sed 's/^0*//'`';' \
-       -e '/_INTERNAL_NAME_/$(if $(shlib),s;_INTERNAL_NAME_;"$(basename $(shlib))";,d)' \
-       -e '/_ORIGINAL_NAME_/$(if $(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)' \
-     $< >$@
-
  # Depend on Makefile.global to force rebuild on re-run of configure.
  win32ver.rc: $(top_builddir)/src/Makefile.global
  endif

-win32ver.o: win32ver.rc
-   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include --include-dir=$(srcdir)
+win32ver.o: $(top_srcdir)/src/port/win32ver.rc
+   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include --include-dir=$(srcdir) -D FILEDESC=$(PGFILEDESC)
-DVFT_APP=$(PGFTYPE) -D_ICO_=$(PGICOSTR) -D_INTERNAL_NAME_=$(if $(shlib),s;_INTERNAL_NAME_;"$(basename $(shlib))";,d)
-D_ORIGINAL_NAME_=$(if$(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)
 


Probably needs some careful checking of the quoting.  But that should be
the right thing in principle.

On 02.09.22 18:57, Andres Freund wrote:
> Is it worth running ninja -t missingdeps as a test? At the time we run tests
> we'll obviously have built and thus collected "real" dependencies, so we would
> have the necessary information to determine whether dependencies are missing.
> I think it'd be fine to do so only for ninja >= 1.11, rather than falling back
> to the llvm python implementation, which is much slower (0.068s vs
> 3.760s). And also because it's not as obvious how to include the python script.
> 
> Alternatively, we could just document that ninja -t missingdeps is worth
> running. Perhaps at the top of the toplevel build.meson file?

In the GNU/make world there is a distinction between "check" and 
"maintainer-check" for this kind of thing.

I think here if we put these kinds of things into a different, what's 
the term, "suite", then that would be a clear way to collect them and be 
able to run them all easily.

On 31.08.22 20:11, Andres Freund wrote:
>> doc/src/sgml/resolv.xsl: I don't understand what this is doing.  Maybe
>> at least add a comment in the file.
> It's only used for building epubs. Perhaps I should extract that into a
> separate patch as well? The relevant section is:
> 
>> #
>> # epub
>> #
>>
>> # This was previously implemented using dbtoepub - but that doesn't seem to
>> # support running in build != source directory (i.e. VPATH builds already
>> # weren't supported).
>> if pandoc.found() and xsltproc.found()
>>    # XXX: Wasn't able to make pandoc successfully resolve entities
>>    # XXX: Perhaps we should just make all targets use this, to avoid repeatedly
>>    # building whole thing? It's comparatively fast though.
>>    postgres_full_xml = custom_target('postgres-full.xml',
>>      input: ['resolv.xsl', 'postgres.sgml'],
>>      output: ['postgres-full.xml'],
>>      depends: doc_generated + [postgres_sgml_valid],
>>      command: [xsltproc, '--path', '@OUTDIR@/', xsltproc_flags,
>>                '-o', '@OUTPUT@', '@INPUT@'],
>>      build_by_default: false,
>>    )
> A noted, I couldn't make pandoc resolve our entities, so I used resolv.xsl
> them, before calling pandoc.
> 
> I'll rename it to resolve-entities.xsl and add a comment.

We can have xmllint do this.  The following gets the epub build working 
with vpath:

diff --git a/doc/src/sgml/Makefile b/doc/src/sgml/Makefile
index 4ae7ca2be7..33b72d03db 100644
--- a/doc/src/sgml/Makefile
+++ b/doc/src/sgml/Makefile
@@ -184,8 +184,12 @@ XSLTPROC_FO_FLAGS += --stringparam img.src.path 
'$(srcdir)/'

  epub: postgres.epub
  postgres.epub: postgres.sgml $(ALLSGML) $(ALL_IMAGES)
-   $(XMLLINT) --noout --valid $<
-   $(DBTOEPUB) -o $@ $<
+   $(XMLLINT) $(XMLINCLUDE) --output tmp.sgml --noent --valid $<
+ifeq ($(vpath_build),yes)
+   $(MKDIR_P) images
+   cp $(ALL_IMAGES) images/
+endif
+   $(DBTOEPUB) -o $@ tmp.sgml


This could also be combined with the idea of the postgres.sgml.valid 
thing you have in the meson patch set.

I'll finish this up and produce a proper patch.

On 07.09.22 09:19, Peter Eisentraut wrote:
> This could also be combined with the idea of the postgres.sgml.valid 
> thing you have in the meson patch set.
> 
> I'll finish this up and produce a proper patch.

Something like this.

This does make the rules more straightforward and avoids repeated 
xmllint calls.  I suppose this also helps writing the meson rules in a 
simpler way.

A possible drawback is that the intermediate postgres-full.xml file is 
 >10MB, but I guess we're past the point where we are worrying about 
that kind of thing.

I don't know if there is any performance difference between xsltproc 
reading one big file versus many smaller files.

On 2022-Sep-06, John Naylor wrote:

> Note that the indentation hasn't changed. My thought there: perltidy
> will be run again next year, at which time it will be part of a listed
> whitespace-only commit. Any objections, since that could confuse
> someone before then?

I think a good plan is to commit the fix without tidy, then commit the
tidy separately, then add the latter commit to .git-blame-ignore-revs.
That avoids leaving the code untidy for a year.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
 Are you not unsure you want to delete Firefox?
       [Not unsure]     [Not not unsure]    [Cancel]
                   http://smylers.hates-software.com/2008/01/03/566e45b2.html

On 04.09.22 20:17, Andres Freund wrote:
> I think, as a followup improvement, we should move gramparse.h to
> src/backend/parser, and stop installing gram.h, gramparse.h. gramparse.h
> already had this note:
> 
>   * NOTE: this file is only meant to be included in the core parsing files,
>   * i.e., parser.c, gram.y, and scan.l.
>   * Definitions that are needed outside the core parser should be in parser.h.
> 
> What do you think?

I found in my notes:

* maybe gram.h and gramparse.h should not be installed

So, yeah. ;-)

On Wed, Sep 7, 2022 at 3:36 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2022-Sep-06, John Naylor wrote:
>
> > Note that the indentation hasn't changed. My thought there: perltidy
> > will be run again next year, at which time it will be part of a listed
> > whitespace-only commit. Any objections, since that could confuse
> > someone before then?
>
> I think a good plan is to commit the fix without tidy, then commit the
> tidy separately, then add the latter commit to .git-blame-ignore-revs.
> That avoids leaving the code untidy for a year.

Okay, done that way. I also made sure we got the same info for error
reporting. It's not identical, but arguably better, going from:

Bareword found where operator expected at (eval 4480) line 3, near "'btree' xxx"
(Missing operator before xxx?)
../../../src/include/catalog/pg_amop.dat: error parsing line 20:

to:

Bareword found where operator expected at (eval 12) line 20, near "'btree' xxx"
(Missing operator before xxx?)
error parsing ../../../src/include/catalog/pg_amop.dat

-- 
John Naylor
EDB: http://www.enterprisedb.com

On 2022-Sep-07, Peter Eisentraut wrote:

> A possible drawback is that the intermediate postgres-full.xml file is
> >10MB, but I guess we're past the point where we are worrying about that
> kind of thing.

I think we are, but maybe mark it .PRECIOUS?  IIUC that would prevent it
from being removed if there's a problem in the other recipes.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"¿Cómo puedes confiar en algo que pagas y que no ves,
y no confiar en algo que te dan y te lo muestran?" (Germán Poo)

On 2022-09-08 14:10:45 +0700, John Naylor wrote:
> Okay, done that way.

Thanks!

On Wed, Sep 7, 2022 at 4:27 PM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
>
> On 04.09.22 20:17, Andres Freund wrote:
> > I think, as a followup improvement, we should move gramparse.h to
> > src/backend/parser, and stop installing gram.h, gramparse.h. gramparse.h
> > already had this note:
> >
> >   * NOTE: this file is only meant to be included in the core parsing files,
> >   * i.e., parser.c, gram.y, and scan.l.
> >   * Definitions that are needed outside the core parser should be in parser.h.
> >
> > What do you think?
>
> I found in my notes:
>
> * maybe gram.h and gramparse.h should not be installed
>
> So, yeah. ;-)

It seems gramparse.h isn't installed now? In any case, here's a patch
to move gramparse to the backend dir and stop symlinking/ installing
gram.h. Confusingly, MSVC didn't seem to copy gram.h to src/include,
so I'm not yet sure how it still managed to build...

-- 
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-08-31 11:11:54 -0700, Andres Freund wrote:
> > If the above are addressed, I think this will be just about at the
> > point where the above patches can be committed.
>
> Woo!

There was a lot less progress over the last ~week than I had hoped. The reason
is that I was trying to figure out the reason for the occasional failures of
ecpg tests getting compiled when building on windows in CI, with msbuild.

I went into many layers of rabbitholes while investigating. Wasting an absurd
amount of time.


The problem:

Occasionally ecpg test files would fail to compile, exiting with -1073741819:
C:\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for
'C:\cirrus\build\meson-private\custom_target.rule'exited with code -1073741819.
[c:\cirrus\build\src\interfaces\ecpg\test\sql\3701597@@twophase.c@cus.vcxproj]

-1073741819 is 0xc0000005, which in turn is STATUS_ACCESS_VIOLATION, i.e. a
segfault. This happens in roughly 1/3 of the builds, but with "streaks" of not
happening and more frequently happening.

However, despite our CI images having a JIT debugger configured (~coredump
handler), no crash report was triggered. The problem never occurs in my
windows VM.


At first I thought that might be because it's an assertion failure or such,
which only causes a dump when a bunch of magic is done (see main.c). But
despite adding all the necessary magic to ecpg.exe, no dump.

Unfortunately, adding debug output reduces the frequency of the issue
substantially.

Eventually I figured out that it's not actually ecpg.exe that is crashing. It
is meson's python wrapper around built binaries as part of the build (for
setting PATH, working directory, without running into cmd.exe issues). A
modified meson wrapper showed that ecpg.exe completes successfully.

The only thing the meson wrapper does after running the command is to call
sys.exit(returncode), and I had printed out the returncode, which is 0.


I looked through a lot of the python code, to see why no crashdump and no
details are forthcoming. There weren't any relevant
SetErrorMode(SEM_NOGPFAULTERRORBOX) calls. I tried to set PYTHONFAULTHANDLER,
but still no stack trace.

Next I suspected that cmd.exe might be crashing and causing the
problem. Modified meson to add 'echo %ERRORLEVEL%' to the msbuild
custombuild. Which indeed shows the STATUS_ACCESS_VIOLATION returncode after
running python. So it's not cmd.exe.


The problem even persisted when replacing meson's sys.exit() with os._exit(),
which indeed just calls _exit().

I tried to reproduce the problem using a python with debugging enabled. The
problem doesn't occur despite quite a few runs.


I found scattered other reports of this problem happening on windows. Went
down a few more rabbitholes. Too boring to repeat here.


At this point I finally figured out that the reason the crash reports don't
happen is that everythin started by cirrus-ci on windows has an errormode of
SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX.

A good bit later I figured out that while cirrus-ci isn't intentionally
setting that, golang does so *unconditionally* on windows:
https://github.com/golang/go/blob/54182ff54a687272dd7632c3a963e036ce03cb7c/src/runtime/signal_windows.go#L14
https://github.com/golang/go/blob/54182ff54a687272dd7632c3a963e036ce03cb7c/src/runtime/os_windows.go#L553
Argh. I should have checked what the error mode is earlier, but this is just
very sneaky.


So I modified meson to change the errormode and tried to reproduce the issue
again, to finally get a stackdump. And tried again. And again. Without a
single relevant failure (I saw tests fail in ways that are discussed on the
list, but that's irrelevant here).

I've run this through enough attempts by now that I'm quite confident that the
problem does not occur when the errormode does not include
SEM_NOOPENFILEERRORBOX. I'll want a few more runs to be certain, but...


Given that the problem appears to happen after _exit() is called, and only
when SEM_NOOPENFILEERRORBOX is not set, it seems likely to be an OS / C
runtime bug. Presumably it's related to something that python does first, but
I don't see how anything could justify crashing only if SEM_NOOPENFILEERRORBOX
is set (rather than the opposite).

I have no idea how to debug this further, given that the problem is quite rare
(can't attach a debugger and wait), only happens when crashdumps are prevented
from happening (so no idea where it crashes) and is made less common by debug
printfs.


So for now the best way forward I can see is to change the error mode for CI
runs. Which is likely a good idea anyway, so we can see crashdumps for
binaries other than postgres.exe (which does SetErrorMode() internally). I
managed to do so by setting CIRRUS_SHELL to a python wrapper around cmd.exe
that does SetErrorMode(). I'm sure there's easier ways, but I couldn't figure
out any.


I'd like to reclaim my time. But I'm afraid nobody will be listening to that
plea...


Greetings,

Andres Freund

On Fri, Sep 9, 2022 at 12:18 PM John Naylor
<john.naylor@enterprisedb.com> wrote:
>
> It seems gramparse.h isn't installed now? In any case, here's a patch
> to move gramparse to the backend dir and stop symlinking/ installing
> gram.h.

Looking more closely at src/include/Makefile, this is incorrect -- all
files in SUBDIRS are copied over. So moving gramparse.h to the backend
will automatically not install it. The explicit install rule for
gram.h was for vpath builds.

CI builds fine. For v2 I only adjusted the commit message. I'll push
in a couple days unless there are objections.
--
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On 2022-09-07 07:00:17 +0200, Peter Eisentraut wrote:
> On 31.08.22 20:11, Andres Freund wrote:
> > > src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
> > > (Note that the latter is also used as an input file for text
> > > substitution.  So having another file named *.in next to it would be
> > > super confusing.)
> > Yea, this stuff isn't great. I think the better solution, both for meson and
> > for configure, would be to move to do all the substitution to the C
> > preprocessor.
>
> Yeah, I think if we can get rid of the evil date-based versioning, then
> this could be done like

> -win32ver.o: win32ver.rc
> -   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include --include-dir=$(srcdir)
> +win32ver.o: $(top_srcdir)/src/port/win32ver.rc
> +   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include --include-dir=$(srcdir) -D
FILEDESC=$(PGFILEDESC)-D VFT_APP=$(PGFTYPE) -D_ICO_=$(PGICOSTR) -D_INTERNAL_NAME_=$(if
$(shlib),s;_INTERNAL_NAME_;"$(basename$(shlib))";,d) -D_ORIGINAL_NAME_=$(if $(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)
 

It tried this and while it works for some places, it doesn't work for all. It
looks like windres uses broken quoting when internally invoking cpp. It
escapes e.g. whitespaces, but it doesn't escape at least < and >. Which
doesn't work well with descriptions like

PGFILEDESC    = "cyrillic <-> mic text conversions"

resulting in this:

strace --string-limit=2000 -f -e execve \
x86_64-w64-mingw32-windres -DPGFILEDESC="cyrillic <-> mic text conversions" -DPGFTYPE=VFT_DLL -DPGNAME=cyrillic_and_mic
-DPGFILEENDING=dll-I../../../../../../src/include -I/home/andres/src/postgresql/src/include
-I/home/andres/src/postgresql/src/include/port/win32 "-I/home/andres/src/postgresql/src/include/port/win32"
-DWIN32_STACK_RLIMIT=4194304-i /home/andres/src/postgresql/src/port/win32ver.rc -o win32ver.o
 
...
[pid 1788987] execve("/bin/sh", ["sh", "-c", "x86_64-w64-mingw32-gcc -E -xc -DRC_INVOKED -DPGFILEDESC=cyrillic\\ <->\\
mic\\text\\ conversions -DPGFTYPE=VFT_DLL -DPGNAME=cyrillic_and_mic -DPGFILEENDING=dll -I../../../../../../src/include
-I/home/andres/src/postgresql/src/include-I/home/andres/src/postgresql/src/include/port/win32
-I/home/andres/src/postgresql/src/include/port/win32-DWIN32_STACK_RLIMIT=4194304
/home/andres/src/postgresql/src/port/win32ver.rc"],0x7ffd47edc790 /* 67 vars */) = 0
 
sh: 1: cannot open -: No such file
[pid 1788987] +++ exited with 2 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1788987, si_uid=1000, si_status=2, si_utime=0, si_stime=0}
---
x86_64-w64-mingw32-windres: preprocessing failed.

given this shoddy quoting, I think it's probably not wise to go down this
path?

We could invoke the preprocessor ourselves, but that requires feeding the
compiler via stdin (otherwise it'll just warn "linker input file unused
because linking not done") and defining -DRC_INVOKED (otherwise there'll be
syntax errors). That feels like too much magic?

Greetings,

Andres Freund

On Sat, Sep 3, 2022 at 2:11 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andres Freund <andres@anarazel.de> writes:
> > 5.14 would be a trivial lift as far as the buildfarm is concerned.
>
> Yeah, that seems like a reasonable new minimum for Perl.  I might
> see about setting up an animal running 5.14.0, just so we can say
> "5.14" in the docs without fine print.

Until such time as that happens, here is a draft to require 5.14.2.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On Tue, Sep 13, 2022 at 5:53 PM John Naylor
<john.naylor@enterprisedb.com> wrote:
>
> Until such time as that happens, here is a draft to require 5.14.2.

As soon as I hit send, it occurred to me that we don't check the perl
version on Windows, since (I seem to recall) 5.8.3 was too old to be
an option on that platform. We'll have to add a new check somewhere.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On 2022-09-12 14:49:50 +0700, John Naylor wrote:
> CI builds fine. For v2 I only adjusted the commit message. I'll push
> in a couple days unless there are objections.

Makes sense to me. Thanks for working on it!

John Naylor <john.naylor@enterprisedb.com> writes:
> On Sat, Sep 3, 2022 at 2:11 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Yeah, that seems like a reasonable new minimum for Perl.  I might
>> see about setting up an animal running 5.14.0, just so we can say
>> "5.14" in the docs without fine print.

> Until such time as that happens, here is a draft to require 5.14.2.

I've just switched longfin to use built-from-source perl 5.14.0.

            regards, tom lane

On Wed, Sep 14, 2022 at 6:47 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> I've just switched longfin to use built-from-source perl 5.14.0.

In that case, here is a quick update with commit message. Not yet any
change for MSVC, but I can put together something later.

Since we're much less willing to support older Windows and Visual
Studio versions, maybe it's low-enough risk defer the check to the
Meson conversion? I understand our MSVC process will then go away much
more quickly than autoconf...

-- 
John Naylor
EDB: http://www.enterprisedb.com

John Naylor <john.naylor@enterprisedb.com> writes:
> On Wed, Sep 14, 2022 at 6:47 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I've just switched longfin to use built-from-source perl 5.14.0.

> In that case, here is a quick update with commit message. Not yet any
> change for MSVC, but I can put together something later.

Looks reasonable just by eyeball, did not test.

> Since we're much less willing to support older Windows and Visual
> Studio versions, maybe it's low-enough risk defer the check to the
> Meson conversion? I understand our MSVC process will then go away much
> more quickly than autoconf...

Agreed --- the MSVC scripts are on a pretty short leash now.
Not clear it's worth fixing them for this point.  If we've
failed to get rid of them by the time v16 release approaches,
maybe it'd be worth doing something then.

            regards, tom lane

On Wed, Sep 14, 2022 at 6:10 AM Andres Freund <andres@anarazel.de> wrote:
>
> On 2022-09-12 14:49:50 +0700, John Naylor wrote:
> > CI builds fine. For v2 I only adjusted the commit message. I'll push
> > in a couple days unless there are objections.
>
> Makes sense to me. Thanks for working on it!

This is done.

-- 
John Naylor
EDB: http://www.enterprisedb.com

On Wed, Sep 14, 2022 at 10:46 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> John Naylor <john.naylor@enterprisedb.com> writes:
> > On Wed, Sep 14, 2022 at 6:47 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> I've just switched longfin to use built-from-source perl 5.14.0.
>
> > In that case, here is a quick update with commit message. Not yet any
> > change for MSVC, but I can put together something later.
>
> Looks reasonable just by eyeball, did not test.
>
> > Since we're much less willing to support older Windows and Visual
> > Studio versions, maybe it's low-enough risk defer the check to the
> > Meson conversion? I understand our MSVC process will then go away much
> > more quickly than autoconf...
>
> Agreed --- the MSVC scripts are on a pretty short leash now.
> Not clear it's worth fixing them for this point.  If we've
> failed to get rid of them by the time v16 release approaches,
> maybe it'd be worth doing something then.

Okay, pushed with no further MSVC changes, after doing a round on CI.

--
John Naylor
EDB: http://www.enterprisedb.com

On 07.09.22 09:53, Peter Eisentraut wrote:
> On 07.09.22 09:19, Peter Eisentraut wrote:
>> This could also be combined with the idea of the postgres.sgml.valid 
>> thing you have in the meson patch set.
>>
>> I'll finish this up and produce a proper patch.
> 
> Something like this.
> 
> This does make the rules more straightforward and avoids repeated 
> xmllint calls.  I suppose this also helps writing the meson rules in a 
> simpler way.

committed this

On 08.09.22 09:42, Alvaro Herrera wrote:
> On 2022-Sep-07, Peter Eisentraut wrote:
> 
>> A possible drawback is that the intermediate postgres-full.xml file is
>>> 10MB, but I guess we're past the point where we are worrying about that
>> kind of thing.
> 
> I think we are, but maybe mark it .PRECIOUS?  IIUC that would prevent it
> from being removed if there's a problem in the other recipes.

I don't think .PRECIOUS is the right tool here.  There are existing uses 
of .SECONDARY in doc/src/sgml/Makefile; I integrated my patch there.

Hi,

Attached is v13 of the meson patchset. The biggest changes are:

- fix for the occasional ecpg.c crashes - which turned out to be crashes of
  python, which in turn likely are due to a bug in the windows CRT

- split out and improved the patch to add resource files on windows. This
  doesn't yet add them to all binaries, but I think the infrastructure looks
  better now, and there's no duplicated win32ver.rc anymore.

- several rebasing adjustments, most notably the parser stuff and the
  introduction of postgresql-full.xml

- improved structure of docs, based on Peter's review (Samay)

- generation of proper dependencies for xmllint/xsltproc, by parsing xsltproc
  --load-trace. Previously the meson build didn't rebuild docs properly (meson
  doesn't have "glob-style" dependencies). (Bilal)

- numerous small improvements and fixes

- added a patch to drop DLLTOOL/DLLWRAP from configure.ac / Makefile.global.in
  - we've removed the use of them in 2014. This way the pgxs emulation doesn't
  need to care.

- noticed that libpgport.a had and needed a dependency on errcodes.h - that
  seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
  postgres.h - which seems wrong. So I added a patch changing them to include
  c.h.


One thing I just realized is that the existing autoconf/make and
src/tools/msvc buildsystems don't generate static libraries for e.g. libpq. So
far the meson build generates both static and shared libraries on windows
too.

Meson solves the naming conflict that presumably lead to us not generating
static libraries on windows by naming the link library for dlls's differently
than static libraries.

I'm inclined to build the static lib on windows as long as we do it on other
platforms.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I'm inclined to build the static lib on windows as long as we do it on other
> platforms.

Maybe I spent too much time working for Red Hat, but I'm kind of
unhappy that we build static libraries at all.  They are maintenance
hazards and therefore security hazards by definition, because if
you find a problem in $package_x you will have to find and rebuild
every other package that has statically-embedded code from $package_x.
So Red Hat has, or least had, a policy against packages exporting
such libraries.

I realize that there are people for whom other considerations outweigh
that, but I don't think that we should install static libraries by
default.  Long ago it was pretty common for configure scripts to
offer --enable-shared and --enable-static options ... should we
resurrect that?

            regards, tom lane

Hi,

On 2022-09-15 01:10:16 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I'm inclined to build the static lib on windows as long as we do it on other
> > platforms.
> 
> Maybe I spent too much time working for Red Hat, but I'm kind of
> unhappy that we build static libraries at all.

Yea, I have been wondering about that too.

Oddly enough, given our current behaviour, the strongest case for static
libraries IMO is on windows, due to the lack of a) rpath b) a general library
search path.

Peter IIRC added the static libraries to the meson port just to keep the set
of installed files the same, which makes sense.


> They are maintenance hazards and therefore security hazards by definition,
> because if you find a problem in $package_x you will have to find and
> rebuild every other package that has statically-embedded code from
> $package_x.  So Red Hat has, or least had, a policy against packages
> exporting such libraries.

It obviously is a bad idea for widely used system packages. I think there are
a few situations, e.g. a downloadable self-contained and relocatable
application, where shared libraries provide less of a benefit.


> I realize that there are people for whom other considerations outweigh
> that, but I don't think that we should install static libraries by
> default.  Long ago it was pretty common for configure scripts to
> offer --enable-shared and --enable-static options ... should we
> resurrect that?

It'd be easy enough. I don't really have an opinion on whether it's worth
having the options. I think most packaging systems have ways of not including
files even if $software installs them.

Greetings,

Andres Freund

On Thu, Sep 15, 2022 at 2:26 PM Andres Freund <andres@anarazel.de> wrote:
> - noticed that libpgport.a had and needed a dependency on errcodes.h - that
>   seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
>   postgres.h - which seems wrong. So I added a patch changing them to include
>   c.h.

Oops.  +1

GCC 12 produces a bunch of warnings by default with meson, and that
turned out to be because the default optimisation level is -O3.
That's a change from the make build, which uses -O2.  Should we set a
default of 2, or is there some meson-way-of-doing-things reason why
not?

Hi,

On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> GCC 12 produces a bunch of warnings by default with meson, and that
> turned out to be because the default optimisation level is -O3.
> That's a change from the make build, which uses -O2.  Should we set a
> default of 2, or is there some meson-way-of-doing-things reason why
> not?

We can change the defaults - the only downside is that there's a convenience
setting 'buildtype' (debug, debugoptimized, release, minsize, custom, plain)
that changes multiple settings (optimization level, amount of debugging
information) and that doesn't work as nicely if you change the default
compiler optimization setting.

They made a similar discovery as us, deriving the defaults of settings based
on other settings quickly can become confusing. I think they're looking at how
to make that UI a bit nicer.

I'd prefer to defer fine-tuning the default settings till a bunch of this has
gone in, but I won't insist on that course.

Their default warning flags passed to compilers trigger a bunch of warnings in
our build (irrespective of -O*), so I lowered the warning level. But I think
their set of settings likely is sensible, an we should just disable a bunch of
warnings we don't care about. But I haven't done that for now, to keep the set
of warning flags the same between meson and autoconf.

Greetings,

Andres Freund

Hi,

On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund <andres@anarazel.de> wrote:
> > - noticed that libpgport.a had and needed a dependency on errcodes.h - that
> >   seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
> >   postgres.h - which seems wrong. So I added a patch changing them to include
> >   c.h.
> 
> Oops.  +1

Looks like this has been the case since
0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
should backpatch the "fix"?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
>> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund <andres@anarazel.de> wrote:
>>> - noticed that libpgport.a had and needed a dependency on errcodes.h - that
>>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
>>> postgres.h - which seems wrong. So I added a patch changing them to include
>>> c.h.

>> Oops.  +1

> Looks like this has been the case since
> 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
> should backpatch the "fix"?

+1, those files have no business including all of postgres.h

            regards, tom lane

Hi,

On 2022-09-16 16:22:35 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> >> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund <andres@anarazel.de> wrote:
> >>> - noticed that libpgport.a had and needed a dependency on errcodes.h - that
> >>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
> >>> postgres.h - which seems wrong. So I added a patch changing them to include
> >>> c.h.
> 
> >> Oops.  +1
> 
> > Looks like this has been the case since
> > 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
> > should backpatch the "fix"?
> 
> +1, those files have no business including all of postgres.h

Done.

I've been wondering whether we should protect against this kind of issue on
the buildsystem level. Whenever building frontend code, add something like
-DBUILDING_FRONTEND, and error out if postgres.h is included without going
through postgres_fe.h.

Greetings,

Andres Freund

On 15.09.22 04:26, Andres Freund wrote:
> Attached is v13 of the meson patchset. The biggest changes are:

Did something about warning flags change from the previous patch set?  I 
see it's building with -Wextra now, which combined with -Werror causes 
the build to fail for me.  I have never encountered that with any of the 
previous patch sets.

Hi,

On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:
>On 15.09.22 04:26, Andres Freund wrote:
>> Attached is v13 of the meson patchset. The biggest changes are:
>
>Did something about warning flags change from the previous patch set?  I see it's building with -Wextra now, which
combinedwith -Werror causes the build to fail for me.  I have never encountered that with any of the previous patch
sets.

In older versions of the patch the default warning level was set to include Wextra, and I had added my local flags to
suppressuninteresting warnings. Comparing the warning flags I reduced the warning level and removed the suppressing
flags- but changing default options only affects new build trees. To change existing ones do meson configure
-Dwarning_level=1
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On 19.09.22 02:29, Andres Freund wrote:
> Hi,
> 
> On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:
>> On 15.09.22 04:26, Andres Freund wrote:
>>> Attached is v13 of the meson patchset. The biggest changes are:
>>
>> Did something about warning flags change from the previous patch set?  I see it's building with -Wextra now, which
combinedwith -Werror causes the build to fail for me.  I have never encountered that with any of the previous patch
sets.
> 
> In older versions of the patch the default warning level was set to include Wextra, and I had added my local flags to
suppressuninteresting warnings. Comparing the warning flags I reduced the warning level and removed the suppressing
flags- but changing default options only affects new build trees. To change existing ones do meson configure
-Dwarning_level=1

Ok that was the reason.  It works now.

IMO, the following commits are ready to be pushed now:

b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in
979f26889544 Don't hardcode tmp_check/ as test directory for tap tests
9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR
6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson
7054861f0fef meson: prereq: Add src/tools/gen_export.pl
1aa586f2921c meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build
5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in *p{read,write}*.c
1939bdcfbfea meson: Add meson based buildsystem

Hi,

On 2022-09-19 05:25:59 -0400, Peter Eisentraut wrote:
> IMO, the following commits are ready to be pushed now:

Slowly working through them.


To have some initial "translation" for other developers I've started a wiki
page with a translation table. Still very WIP:
https://wiki.postgresql.org/wiki/Meson

For now, a bit of polishing aside, I'm just planning to add a minimal
explanation of what's happening, and a reference to this thread.


> b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in
> 979f26889544 Don't hardcode tmp_check/ as test directory for tap tests
> 9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR
> 6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson
> 5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in *p{read,write}*.c

Done


> 7054861f0fef meson: prereq: Add src/tools/gen_export.pl

This one I'm planning to merge with the "main" commit, given there's no other user.

Greetings,

Andres Freund

Hi,

On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> To have some initial "translation" for other developers I've started a wiki
> page with a translation table. Still very WIP:
> https://wiki.postgresql.org/wiki/Meson
> 
> For now, a bit of polishing aside, I'm just planning to add a minimal
> explanation of what's happening, and a reference to this thread.

I added installation instructions for meson for a bunch of platforms, but
failed to figure out how to do so in a rhel9 container. I don't have a rhel
subscription, and apparently the repos with developer tools now require a
subscription. Great way to make it easy for projects to test anything on RHEL.

Greetings,

Andres Freund

Hi,

On 2022-09-21 09:52:48 +0700, John Naylor wrote:
> On Wed, Sep 21, 2022 at 7:11 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> > > To have some initial "translation" for other developers I've started a
> wiki
> > > page with a translation table. Still very WIP:
> > > https://wiki.postgresql.org/wiki/Meson
> > >
> > > For now, a bit of polishing aside, I'm just planning to add a minimal
> > > explanation of what's happening, and a reference to this thread.
> >
> > I added installation instructions for meson for a bunch of platforms, but
> 
> Small typo: The homebrew section is still labeled with "find MacPorts
> libraries".

Thanks, fixed. I wrote these blindly, so there's probably more wrong than this
- although Thomas was helpful enough to provide some information / testing.

Greetings,

Andres Freund

Hi,

On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> On 2022-09-19 05:25:59 -0400, Peter Eisentraut wrote:
> > IMO, the following commits are ready to be pushed now:
> 
> Slowly working through them.

I've attached an updated version of the main meson commit (and pushed all of
them to my git tree obviously). Changes:

- Added a longer commit message
- Stopped building doc/src/sgml/postgres-full.xml by default - somehow I
  thought we did so by default for the autoconf build, but that's not the
  case. Thomas noticed that that was extremely slow on one of his machines,
  which turns out to be because it's downloading the dtd's.
- Added a missing dependency on check_rules.pl's result, lost that in a
  cleanup, oops
- Fixed a few typos, via codespell

I'm planning to commit this today, unless somebody wants to argue against
that.


After that I am planning to split the "ci" commit so that it converts a few of
the CI tasks to use meson, without adding all the other platforms I added for
development. I think that's important to get in soon, given that it'll
probably take a bit until the buildfarm grows meson coverage and because it
provides cfbot coverage which seems important for now as well.

I think we should:

- convert windows to build with ninja - it builds faster, runs all tests,
  parallelizes tests. That means that msbuild based builds don't have coverage
  via CI / cfbot, but we don't currently have the resources to test both.
- add a linux build using meson, we currently can afford building both with
  autoconf and meson for linux

I'm less clear on whether we should convert macos / freebsd to meson at this
point?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> I think we should:

> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

Check.  The sooner we can get rid of the custom MSVC scripts, the better,
because now we'll be on the hook to maintain *three* build systems.

> - add a linux build using meson, we currently can afford building both with
>   autoconf and meson for linux

Right.

> I'm less clear on whether we should convert macos / freebsd to meson at this
> point?

We certainly could debug/polish the meson stuff just on linux and windows
for now, but is there a reason to wait on the others?

            regards, tom lane

Hi,

On 2022-09-21 13:56:37 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I think we should:
> 
> > - convert windows to build with ninja - it builds faster, runs all tests,
> >   parallelizes tests. That means that msbuild based builds don't have coverage
> >   via CI / cfbot, but we don't currently have the resources to test both.
> 
> Check.  The sooner we can get rid of the custom MSVC scripts, the better,
> because now we'll be on the hook to maintain *three* build systems.

Agreed. I think the only "major" missing thing is the windows resource file
generation stuff, which is mostly done in one of the "later" commits. Also
need to test a few more of the optional dependencies (ICU, gettext, ...) on
windows (I did test zlib, lz4, zstd). And of course get a bit of wider
exposure than "just me and CI".


> > I'm less clear on whether we should convert macos / freebsd to meson at this
> > point?
> 
> We certainly could debug/polish the meson stuff just on linux and windows
> for now, but is there a reason to wait on the others?

No - freebsd and macos have worked in CI for a long time. I was wondering
whether we want more coverage for autoconf in CI, but thinking about it
futher, it's more important to have the meson coverage.

Greetings,

Andres Freund

On Wed, Sep 21, 2022 at 09:46:30AM -0700, Andres Freund wrote:
> I think we should:
> 
> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

+1

If multiple Windows (or other) tasks are going to exist, I think they
should have separate "ci-os-only" conditions, like windows-msvc,
windows-ninja, ...  It should be possible to run only one.

Hi,

On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
> After that I am planning to split the "ci" commit so that it converts a few of
> the CI tasks to use meson, without adding all the other platforms I added for
> development. I think that's important to get in soon, given that it'll
> probably take a bit until the buildfarm grows meson coverage and because it
> provides cfbot coverage which seems important for now as well.
> 
> I think we should:
> 
> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

I was working on that and hit an issue that took me a while to resolve: Once I
tested only the "main" meson commit plus CI the windows task was running out
of memory. There was an outage of the CI provider at the same time, so I first
blamed it on that. But it turns out to be "legitimately" high memory usage
related to debug symbols - the only reason CI didn't show that before was that
it's incidentally fixed as a indirect consequence of using precompiled
headers, in a later commit. Argh.  It can also be fixed by the option required
to use ccache at some point, so I'll do that for now.

Greetings,

Andres Freund

Hi,

On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
> I'm planning to commit this today, unless somebody wants to argue against
> that.

And done!

Changes:
- fixed a few typos (thanks Thomas)
- less duplication in the CI tasks
- removed an incomplete implementation of the target for abbrevs.txt - do we
  even want to have that?
- plenty hand wringing on my part


I also rebased my meson git tree, which still has plenty additional test
platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7,
opensuse tumbleweed), but without the autoconf versions of those targets.  I
also added a commit that translates most of the CompilerWarnings task to
meson.  Still need to add a headerscheck / cpluspluscheck target.

Greetings,

Andres Freund

On 2022-09-22 Th 01:57, Andres Freund wrote:
> Hi,
>
> On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
>> I'm planning to commit this today, unless somebody wants to argue against
>> that.
> And done!
>
> Changes:
> - fixed a few typos (thanks Thomas)
> - less duplication in the CI tasks
> - removed an incomplete implementation of the target for abbrevs.txt - do we
>   even want to have that?
> - plenty hand wringing on my part
>
>
> I also rebased my meson git tree, which still has plenty additional test
> platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7,
> opensuse tumbleweed), but without the autoconf versions of those targets.  I
> also added a commit that translates most of the CompilerWarnings task to
> meson.  Still need to add a headerscheck / cpluspluscheck target.
>

Great. Now I'll start on buildfarm support. Given my current
commitments, this will take me a while, but I hope to have a working
client by about the beginning of November.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andres Freund <andres@anarazel.de> writes:
> On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
>> I'm planning to commit this today, unless somebody wants to argue against
>> that.

> And done!

Yay!

Initial reports from the cfbot are mostly promising, but there are a few
patches where all the meson builds fail while all the autoconf ones pass,
so there's something for you to look at.  So far CF entries 3464, 3733,
3771, 3808 look that way.

            regards, tom lane

On 2022-Sep-22, Tom Lane wrote:

> Initial reports from the cfbot are mostly promising, but there are a few
> patches where all the meson builds fail while all the autoconf ones pass,
> so there's something for you to look at.  So far CF entries 3464, 3733,
> 3771, 3808 look that way.

Hmm, but those patches add files, which means they're now outdated: they
need to add these files to the respective meson.build file.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"La experiencia nos dice que el hombre peló millones de veces las patatas,
pero era forzoso admitir la posibilidad de que en un caso entre millones,
las patatas pelarían al hombre" (Ijon Tichy)

Hi,

On 2022-09-22 16:56:57 +0200, Alvaro Herrera wrote:
> On 2022-Sep-22, Tom Lane wrote:
> > Initial reports from the cfbot are mostly promising, but there are a few
> > patches where all the meson builds fail while all the autoconf ones pass,
> > so there's something for you to look at.  So far CF entries 3464, 3733,
> > 3771, 3808 look that way.
> 
> Hmm, but those patches add files, which means they're now outdated: they
> need to add these files to the respective meson.build file.

Yea, I looked through all of these and they all need need a simple addition of
a file to be built or installed.

Greetings,

Andres Freund

Hi,

On 2022-09-22 04:29:15 -0400, Andrew Dunstan wrote:
> Great. Now I'll start on buildfarm support. Given my current
> commitments, this will take me a while, but I hope to have a working
> client by about the beginning of November.

Great! Let me know if there's something I can do to help.

Greetings,

Andres Freund

I gave the meson build system a try, and it seems to work nicely.  It
didn't take long at all to adapt my workflow.

A few notes from my experience:

* I'm using an Ubuntu-based distribution, and the version of meson that apt
installed was not new enough for Postgres.  I ended up cloning meson [0]
and using the newest tag.  This is no big deal.

* The installed binaries were unable to locate libraries like libpq.  I
ended up setting the extra_lib_dirs option to the directory where these
libraries were installed to fix this.  This one is probably worth
investigating further.

* meson really doesn't like it when there are things leftover from
configure/make.  Whenever I switch from make to meson, I have to run 'make
maintainer-clean'.

Otherwise, all of my usual build options, ccache, etc. are working just
like before.  Nice work!

[0] https://github.com/mesonbuild/meson

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On Thu, Sep 22, 2022 at 1:05 PM Nathan Bossart <nathandbossart@gmail.com> wrote:
> Otherwise, all of my usual build options, ccache, etc. are working just
> like before.  Nice work!

+1

Is it generally recommended that individual hackers mostly switch over
to Meson for their day to day work soon? I'm guessing that this
question doesn't really have a clear answer yet, but thought I'd ask,
just in case.

--
Peter Geoghegan

Hi,

On 2022-09-22 13:05:33 -0700, Nathan Bossart wrote:
> I gave the meson build system a try, and it seems to work nicely.  It
> didn't take long at all to adapt my workflow.
> 
> A few notes from my experience:
> 
> * I'm using an Ubuntu-based distribution, and the version of meson that apt
> installed was not new enough for Postgres.  I ended up cloning meson [0]
> and using the newest tag.  This is no big deal.

I assume this is 20.04 LTS? If so, we're missing it by one version of meson
currently. There's unfortunately a few features that'd be a bit painful to not
have.


> * The installed binaries were unable to locate libraries like libpq.  I
> ended up setting the extra_lib_dirs option to the directory where these
> libraries were installed to fix this.  This one is probably worth
> investigating further.

I think that should be "fixed" in a later commit in the meson tree - any
chance you could try that?

https://github.com/anarazel/postgres/tree/meson


> * meson really doesn't like it when there are things leftover from
> configure/make.  Whenever I switch from make to meson, I have to run 'make
> maintainer-clean'.

Yes. I recommend building out-of-tree with autoconf as well.


> Otherwise, all of my usual build options, ccache, etc. are working just
> like before.  Nice work!

Cool!

Thanks for testing,

Andres Freund

Hi,

On 2022-09-22 13:21:28 -0700, Peter Geoghegan wrote:
> Is it generally recommended that individual hackers mostly switch over
> to Meson for their day to day work soon? I'm guessing that this
> question doesn't really have a clear answer yet, but thought I'd ask,
> just in case.

It'll probably depend on who you ask ;)

I'm likely the most biased person on this, but for me the reliable incremental
builds and the readability of the test output are big enough wins that the
answer is pretty clear... Doesn't hurt that running all tests is faster too.


The currently existing limitations are imo mostly around making it usable for
production, particularly on windows.


time to run all tests (cassert, -Og), in a fully built tree:

make:

time make -j48 -s -Otarget check-world
real    2m44.206s
user    6m29.121s
sys    1m54.069s

time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4'
real    1m1.577s
user    7m32.579s
sys    2m17.767s


meson:

time meson test
real    0m42.178s
user    7m8.533s
sys    2m17.711s


FWIW, I just rebased my older patch to cache and copy initdb during the
tests. The %user saved is impressive enough to pursue it again...

time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4'
real    0m52.655s
user    2m19.504s
sys    1m26.264s

time meson test:

real    0m36.370s
user    2m14.748s
sys    1m36.741s


Greetings,

Andres Freund