Обсуждение: pgsql: aix: No need to use mkldexport when we want to export all symbol

aix: No need to use mkldexport when we want to export all symbols

When building a shared library with exports.txt there's no need to build an
intermediary static library, we can just pass -Wl,-bE:... when generating the
.so.

When building a shared library without exports.txt, there's no need to call
mkldexport.sh to export all symbols, because all symbols are exported anyway,
and we don't need the export file on the import side (like we do for
postgres.imp).

This makes building .so's on aix a lot more similar to building on other
platforms. In particular, we don't create and remove a .a of the same name but
different contents anymore.

Discussion: https://postgr.es/m/20220820174213.d574qde4ptwdzoqz@awork3.anarazel.de

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/fe6a64a58ab3e5bda3aceee2f1ee3e8efdb03865

Modified Files
--------------
src/Makefile.shlib         | 45 ++++++++++++++++++---------------------------
src/makefiles/Makefile.aix |  8 ++------
2 files changed, 20 insertions(+), 33 deletions(-)

Andres Freund <andres@anarazel.de> writes:
> aix: No need to use mkldexport when we want to export all symbols

hmm, early returns not good:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2022-09-10%2003%3A31%3A29

The backend seems to build OK, but our first try to make a .so
falls over:

gmake[3]: Entering directory
'/scratch/nm/farm/xlc64v16/HEAD/pgsql.build/src/backend/utils/mb/conversion_procs/cyrillic_and_mic'
/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432
-qsuppress=1500-010:1506-995-qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264
-qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse-qinfo=nounset
-I../../../../../../src/include-I/home/nm/sw/nopath/icu58.3-64/include
-I/home/nm/sw/nopath/libxml2-64/include/libxml2 -I/home/nm/sw/nopath/uuid-64/include
-I/home/nm/sw/nopath/openldap-64/include-I/home/nm/sw/nopath/icu58.3-64/include -I/home/nm/sw/nopath/libxml2-64/include
-c -o cyrillic_and_mic.o cyrillic_and_mic.c 
/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432
-qsuppress=1500-010:1506-995-qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264
-qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse-qinfo=nounset   -o cyrillic_and_mic.so
cyrillic_and_mic.o -L../../../../../../src/port -L../../../../../../src/common   -L/home/nm/sw/nopath/libxml2-64/lib
-L/home/nm/sw/nopath/icu58.3-64/lib -L/home/nm/sw/nopath/uuid-64/lib -L/home/nm/sw/nopath/openldap-64/lib
-L/home/nm/sw/nopath/icu58.3-64/lib-L/home/nm/sw/nopath/libxml2-64/lib   -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE
-Wl,-bI:../../../../../../src/backend/postgres.imp
ld: 0711-244 ERROR: No csects or exported symbols have been saved.
../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
gmake[3]: *** [cyrillic_and_mic.so] Error 8

            regards, tom lane

Hi,

On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>Andres Freund <andres@anarazel.de> writes:
>> aix: No need to use mkldexport when we want to export all symbols
>
>hmm, early returns not good:
>
>https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2022-09-10%2003%3A31%3A29
>
>The backend seems to build OK, but our first try to make a .so
>falls over:
>
>ld: 0711-244 ERROR: No csects or exported symbols have been saved.
>../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
>gmake[3]: *** [cyrillic_and_mic.so] Error 8

I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hi,

On 2022-09-09 21:16:38 -0700, Andres Freund wrote:
> On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >Andres Freund <andres@anarazel.de> writes:
> >> aix: No need to use mkldexport when we want to export all symbols
> >
> >hmm, early returns not good:
> >
> >https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2022-09-10%2003%3A31%3A29
> >
> >The backend seems to build OK, but our first try to make a .so
> >falls over:
> >
> >ld: 0711-244 ERROR: No csects or exported symbols have been saved.
> >../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
> >gmake[3]: *** [cyrillic_and_mic.so] Error 8
> 
> I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.

A basic build with xlc 16.1 still succeeds building world-bin. Including
obviously cyrillic_and_mic.so

../configure PERL=perl64 --cache ../config-ac-xlc.cache --with-perl CC=/opt/IBM/xlc/16.1.0/bin/xlc_r
CFLAGS="-qmaxmem=33554432-qsuppress=1500-010:1506-995
-qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264
-qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse-qinfo=nounset"
 

with OBJECT_MODE=64 previously set.

I'll try a build closer to Noah's configure options. Most likely candidate
seems --enable-debug.

Greetings,

Andres Freund

Hi,

On 2022-09-09 21:37:49 -0700, Andres Freund wrote:
> On 2022-09-09 21:16:38 -0700, Andres Freund wrote:
> > On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >Andres Freund <andres@anarazel.de> writes:
> > >> aix: No need to use mkldexport when we want to export all symbols
> > >
> > >hmm, early returns not good:
> > >
> > >https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2022-09-10%2003%3A31%3A29
> > >
> > >The backend seems to build OK, but our first try to make a .so
> > >falls over:
> > >
> > >ld: 0711-244 ERROR: No csects or exported symbols have been saved.
> > >../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
> > >gmake[3]: *** [cyrillic_and_mic.so] Error 8
> > 
> > I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.
> 
> A basic build with xlc 16.1 still succeeds building world-bin. Including
> obviously cyrillic_and_mic.so
> 
> ../configure PERL=perl64 --cache ../config-ac-xlc.cache --with-perl CC=/opt/IBM/xlc/16.1.0/bin/xlc_r
CFLAGS="-qmaxmem=33554432-qsuppress=1500-010:1506-995
-qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264
-qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse-qinfo=nounset"
 
> 
> with OBJECT_MODE=64 previously set.
> 
> I'll try a build closer to Noah's configure options. Most likely candidate
> seems --enable-debug.

Hm. In a new tree it doesn't work with HEAD. But it did previously survive a
gmake clean && gmake world-bin. I don't see why the configure cache or
existing buildtree should play a role here, given a make clean?

Either way, the symbol visibility patch from
https://postgr.es/m/20220901211935.k6v7mt3mjhnfbhnr%40awork3.anarazel.de
repairs the build, even with a new build tree.

Looking at that patch I don't like this bit overly much:
+if test -n "${CFLAGS_SL_MODULE}"; then
+  AC_DEFINE([HAVE_VISIBILITY_ATTRIBUTE], 1,
+            [Define to 1 if your compiler knows the visibility("hidden") attribute.])
+fi

Seems too fragile, somebody could test for another attribute to add to
CFLAGS_SL_MODULE and it'd break. I'm inclined to just set a
pgac_have_visibility_attr in the two branches and set it from
pgac_cv_prog_CC_cflags__fvisibility_hidden
pgac_cv_prog_CC_cflags__qvisibility_hidden
?

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm? I guess alternatively we could revert the mkldexport patch,
apply the -qvisibility patch and then apply the mkldexport thing again
afterwards?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> It seem worth applying the -qvisibility patch and seeing whether that fixes
> the buildfarm?

Worth a try.

> I guess alternatively we could revert the mkldexport patch,
> apply the -qvisibility patch and then apply the mkldexport thing again
> afterwards?

Seems pretty academic, unless -qvisibility doesn't fix it.

            regards, tom lane

Hi,

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > the buildfarm?
> 
> Worth a try.

Done now.

Looks like we also could get away with -Wl,-bexpall or such, but of course
selective exporting is preferrable. Just noting that in case an older xlc
version makes trouble.

Greetings,

Andres Freund

Hi,

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:
> On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> > Andres Freund <andres@anarazel.de> writes:
> > > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > > the buildfarm?
> > 
> > Worth a try.
> 
> Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

Greetings,

Andres Freund

Hi,

On 2022-09-10 01:19:44 -0700, Andres Freund wrote:
> On 2022-09-09 22:57:36 -0700, Andres Freund wrote:
> > On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> > > Andres Freund <andres@anarazel.de> writes:
> > > > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > > > the buildfarm?
> > >
> > > Worth a try.
> >
> > Done now.
>
> Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
> still happy. Still need to wait for the older xlc's (mandril, hornet)
> though...

At least hornet isn't happy:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2022-09-10%2021%3A44%3A00

Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:

https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS214-162

> -qvisibility option specifies visibility attributes for entities. Entity
>  visibility attributes describe whether and how entities defined in one
>  module can be referenced or used in other modules. Visibility attributes
>  affect entities with external linkage only, and cannot increase the
>  visibility of other entities.


Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.  We could also try to fall back to
using -Wl,b,expall for extension modules.

Greetings,

Andres Freund

On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:
> On 2022-09-10 01:19:44 -0700, Andres Freund wrote:
> > On 2022-09-09 22:57:36 -0700, Andres Freund wrote:
> > > On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> > > > Andres Freund <andres@anarazel.de> writes:
> > > > > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > > > > the buildfarm?
> > > >
> > > > Worth a try.
> > >
> > > Done now.
> >
> > Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
> > still happy. Still need to wait for the older xlc's (mandril, hornet)
> > though...
> 
> At least hornet isn't happy:
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2022-09-10%2021%3A44%3A00
> 
> Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
>
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS214-162

Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-12%2006%3A24%3A53
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-10%2018%3A09%3A11

Both runs had the latest commits discussed above.  tern (gcc32) passed, but it
doesn't use --with-perl.

> > -qvisibility option specifies visibility attributes for entities. Entity
> >  visibility attributes describe whether and how entities defined in one
> >  module can be referenced or used in other modules. Visibility attributes
> >  affect entities with external linkage only, and cannot increase the
> >  visibility of other entities.
> 
> 
> Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
> we should increase our requirement to 13.1 rather than revert back to the
> higher complexity way of building modules.

Perhaps.  That would demolish two buildfarm members.  What does PostgreSQL
lose if you revert fe6a64a?

> We could also try to fall back to
> using -Wl,b,expall for extension modules.

I have no knowledge of that option.

Hi,

On 2022-09-12 20:38:45 -0700, Noah Misch wrote:
> On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:
> > On 2022-09-10 01:19:44 -0700, Andres Freund wrote:
> > > On 2022-09-09 22:57:36 -0700, Andres Freund wrote:
> > > > On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> > > > > Andres Freund <andres@anarazel.de> writes:
> > > > > > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > > > > > the buildfarm?
> > > > >
> > > > > Worth a try.
> > > >
> > > > Done now.
> > >
> > > Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
> > > still happy. Still need to wait for the older xlc's (mandril, hornet)
> > > though...
> >
> > At least hornet isn't happy:
> >
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2022-09-10%2021%3A44%3A00
> >
> > Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
> >
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS214-162
>
> Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-12%2006%3A24%3A53
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-10%2018%3A09%3A11
>
> Both runs had the latest commits discussed above.  tern (gcc32) passed, but it
> doesn't use --with-perl.

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

What is "perl64-for-gcc"?


> > > -qvisibility option specifies visibility attributes for entities. Entity
> > >  visibility attributes describe whether and how entities defined in one
> > >  module can be referenced or used in other modules. Visibility attributes
> > >  affect entities with external linkage only, and cannot increase the
> > >  visibility of other entities.
> >
> >
> > Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
> > we should increase our requirement to 13.1 rather than revert back to the
> > higher complexity way of building modules.
>
> Perhaps.  That would demolish two buildfarm members.

Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
available?


> What does PostgreSQL lose if you revert fe6a64a?

A somewhat readable Makefile.shlib. The prior state is complicated, specific
to AIX. I've several times spent quite a bit of time understanding it - not
helped by the complete lack of comments. It's not frequent enough to stay in
my brain unfortunately, so I'll have to relearn it again next time. And we'll
need to continue supporting Makefile.shlib for quite a while, so ...

I do not understand what value it has to run an ancient compiler on a very
niche, quite different, OS. I can see some value to supporting a larger number
of operating systems and compiles - but I can't see any sort of benefit in
also supporting old versions of those compilers.

Greetings,

Andres Freund

On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:
> On 2022-09-12 20:38:45 -0700, Noah Misch wrote:
> > On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:
> > > On 2022-09-10 01:19:44 -0700, Andres Freund wrote:
> > > > On 2022-09-09 22:57:36 -0700, Andres Freund wrote:
> > > > > On 2022-09-10 01:32:52 -0400, Tom Lane wrote:
> > > > > > Andres Freund <andres@anarazel.de> writes:
> > > > > > > It seem worth applying the -qvisibility patch and seeing whether that fixes
> > > > > > > the buildfarm?
> > > > > >
> > > > > > Worth a try.
> > > > >
> > > > > Done now.
> > > >
> > > > Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
> > > > still happy. Still need to wait for the older xlc's (mandril, hornet)
> > > > though...
> > >
> > > At least hornet isn't happy:
> > >
> > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2022-09-10%2021%3A44%3A00
> > >
> > > Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
> > >
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS214-162
> >
> > Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-12%2006%3A24%3A53
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-09-10%2018%3A09%3A11
> >
> > Both runs had the latest commits discussed above.  tern (gcc32) passed, but it
> > doesn't use --with-perl.
> 
> I'll check it out tomorrow. I configured perl on the other AIX gcc compile
> farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
> they seem to be down?

They're up.

> What is "perl64-for-gcc"?

===
#! /bin/sh

real_perl=perl64

case $* in
    '-MExtUtils::Embed -e ldopts' | \
    '-MConfig -e print $Config{ccdlflags}')
        $real_perl "$@" | sed 's/-b/-Wl,-b/g' ;;
    *)
        exec $real_perl "$@" ;;
esac
===

> > > > -qvisibility option specifies visibility attributes for entities. Entity
> > > >  visibility attributes describe whether and how entities defined in one
> > > >  module can be referenced or used in other modules. Visibility attributes
> > > >  affect entities with external linkage only, and cannot increase the
> > > >  visibility of other entities.
> > >
> > >
> > > Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
> > > we should increase our requirement to 13.1 rather than revert back to the
> > > higher complexity way of building modules.
> >
> > Perhaps.  That would demolish two buildfarm members.
> 
> Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
> available?

I don't see 13.x, but I might not know where to look.

> > What does PostgreSQL lose if you revert fe6a64a?
> 
> A somewhat readable Makefile.shlib. The prior state is complicated, specific
> to AIX. I've several times spent quite a bit of time understanding it - not
> helped by the complete lack of comments. It's not frequent enough to stay in
> my brain unfortunately, so I'll have to relearn it again next time. And we'll
> need to continue supporting Makefile.shlib for quite a while, so ...

I'm voting for a revert, then.  Code beautification is nice when it just
works, but desupporting this compiler to land this particular code
beautification makes the net value go negative for me.  I can understand your
feeling differently based on your recent experiences.

Hi,

On 2022-09-12 23:11:23 -0700, Andres Freund wrote:
> I'll check it out tomorrow. I configured perl on the other AIX gcc compile
> farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
> they seem to be down?

Seems to just have been a temporary connectivity issue.

After battling a bit with bison flex (system m4 seems incompatible with system
flex?), solved by using your binaries, I suceeded building and testing plperl
on 346990ae2e

~/src/postgres/configure FLEX=/home/nm/bin/flex BISON=/home/nm/sw/bison/bin/bison CXX='g++ -maix64' CC='gcc -maix64'
PERL=perl64--cache ../config-ac-gcc.cache --enable-cassert --enable-debug --with-perl 

gmake -j8 -s world-bin && gmake -j8 -s checkprep && gmake -j8 -s temp-install
cd src/pl/plperl/
gmake check NO_TEMP_INSTALL=1
...
======================
 All 13 tests passed.
======================

"my" linking of plperl.so:

gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute-Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard-Wno-format-truncation -Wno-stringop-truncation -g -O2  -fvisibility=hidden -o plperl.so
plperl.oSPI.o Util.o  -L../../../src/port -L../../../src/common
-Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/'
-Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden
-L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE-lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads
-lc-Wl,-bI:../../../src/backend/postgres.imp 

sungazer:
wrap-gcc-8.3 -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix64 -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement-Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3
-Wcast-function-type-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation
-Wno-stringop-truncation-g -O2  -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o  -L../../../src/port
-L../../../src/common  -L/home/nm/sw/nopath/libxml2-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib
-L/home/nm/sw/nopath/uuid-64/lib-L/home/nm/sw/nopath/openldap-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib
-L/home/nm/sw/nopath/libxml2-64/lib
-Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/home/nm/sw/nopath/libxml2-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/uuid-64/lib:/home/nm/sw/nopath/openldap-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/libxml2-64/lib:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/'
-Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden
-Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp
-Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp-Wl,-brtl -Wl,-bdynamic -Wl,-b64
-L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE-lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads
-lc-Wl,-bI:../../../src/backend/postgres.imp 

I suspect the problem is the -Wl,-bE:/.../perl.exp - that'll obviously
restrict the exports of plperl.so to the ones of perl, which obviously is
wrong.  I would have assumed that Peter's recent plperl changes should have
gotten rid of the -Wl,-bE:?


Greetings,

Andres Freund

Hi,

On 2022-09-12 23:39:04 -0700, Noah Misch wrote:
> On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:
> > On 2022-09-12 20:38:45 -0700, Noah Misch wrote:
> > > On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:
> > > Both runs had the latest commits discussed above.  tern (gcc32) passed, but it
> > > doesn't use --with-perl.
> >
> > I'll check it out tomorrow. I configured perl on the other AIX gcc compile
> > farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
> > they seem to be down?
>
> They're up.

Yep, seems to have been temporary.


> > What is "perl64-for-gcc"?
>
> ===
> #! /bin/sh
>
> real_perl=perl64
>
> case $* in
>     '-MExtUtils::Embed -e ldopts' | \
>     '-MConfig -e print $Config{ccdlflags}')
>         $real_perl "$@" | sed 's/-b/-Wl,-b/g' ;;
>     *)
>         exec $real_perl "$@" ;;
> esac
> ===

I guess that defeats our plperl flags logic - the wrapper shouldn't be needed
anymore today, see my parallel message about the problem likely being caused
by -Wl,-bE:.


> > > > > -qvisibility option specifies visibility attributes for entities. Entity
> > > > >  visibility attributes describe whether and how entities defined in one
> > > > >  module can be referenced or used in other modules. Visibility attributes
> > > > >  affect entities with external linkage only, and cannot increase the
> > > > >  visibility of other entities.
> > > >
> > > >
> > > > Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
> > > > we should increase our requirement to 13.1 rather than revert back to the
> > > > higher complexity way of building modules.
> > >
> > > Perhaps.  That would demolish two buildfarm members.
> >
> > Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
> > available?
>
> I don't see 13.x, but I might not know where to look.

Looks like it's not available on gcc111, just on gcc119, there it's at
/opt/IBM/xlc/13.1.3/bin/xlc


> > > What does PostgreSQL lose if you revert fe6a64a?
> >
> > A somewhat readable Makefile.shlib. The prior state is complicated, specific
> > to AIX. I've several times spent quite a bit of time understanding it - not
> > helped by the complete lack of comments. It's not frequent enough to stay in
> > my brain unfortunately, so I'll have to relearn it again next time. And we'll
> > need to continue supporting Makefile.shlib for quite a while, so ...
>
> I'm voting for a revert, then.  Code beautification is nice when it just
> works, but desupporting this compiler to land this particular code
> beautification makes the net value go negative for me.  I can understand your
> feeling differently based on your recent experiences.

IMO this more than just a beautification - the difference is functional in
nature. I'd agree if it just were a reformating or such, but this isn't that.

I just tested, and with the mlkdexport.sh approach we end up with all symbols
exported (-fvisibility=hidden) in extension .so's - AIX will be the only
platform unable to restrict symbol exports. That's bound to lead to problems
we'll only have on AIX.

The more special-case-y some platform is, the higher the cost of supporting it
is. I'm somewhat ok with supporting niche stuff, but then the people desiring
that support also need to do the work of making that bearable for everyone
else.

Greetings,

Andres Freund

Hi,

On 2022-09-13 00:02:53 -0700, Andres Freund wrote:
> On 2022-09-12 23:11:23 -0700, Andres Freund wrote:
> > I'll check it out tomorrow. I configured perl on the other AIX gcc compile
> > farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
> > they seem to be down?
>
> Seems to just have been a temporary connectivity issue.
>
> After battling a bit with bison flex (system m4 seems incompatible with system
> flex?), solved by using your binaries, I suceeded building and testing plperl
> on 346990ae2e
>
> ~/src/postgres/configure FLEX=/home/nm/bin/flex BISON=/home/nm/sw/bison/bin/bison CXX='g++ -maix64' CC='gcc -maix64'
PERL=perl64--cache ../config-ac-gcc.cache --enable-cassert --enable-debug --with-perl 
>
> gmake -j8 -s world-bin && gmake -j8 -s checkprep && gmake -j8 -s temp-install
> cd src/pl/plperl/
> gmake check NO_TEMP_INSTALL=1
> ...
> ======================
>  All 13 tests passed.
> ======================
>
> "my" linking of plperl.so:
>
> gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute-Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard-Wno-format-truncation -Wno-stringop-truncation -g -O2  -fvisibility=hidden -o plperl.so
plperl.oSPI.o Util.o  -L../../../src/port -L../../../src/common
-Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/'
-Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden
-L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE-lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads
-lc-Wl,-bI:../../../src/backend/postgres.imp 
>
> sungazer:
> wrap-gcc-8.3 -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix64 -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement-Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3
-Wcast-function-type-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation
-Wno-stringop-truncation-g -O2  -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o  -L../../../src/port
-L../../../src/common  -L/home/nm/sw/nopath/libxml2-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib
-L/home/nm/sw/nopath/uuid-64/lib-L/home/nm/sw/nopath/openldap-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib
-L/home/nm/sw/nopath/libxml2-64/lib
-Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/home/nm/sw/nopath/libxml2-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/uuid-64/lib:/home/nm/sw/nopath/openldap-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/libxml2-64/lib:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/'
-Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden
-Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp
-Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp-Wl,-brtl -Wl,-bdynamic -Wl,-b64
-L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE-lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads
-lc-Wl,-bI:../../../src/backend/postgres.imp 
>
> I suspect the problem is the -Wl,-bE:/.../perl.exp - that'll obviously
> restrict the exports of plperl.so to the ones of perl, which obviously is
> wrong.  I would have assumed that Peter's recent plperl changes should have
> gotten rid of the -Wl,-bE:?

The spurious -Wl,-bE's are the cause of all these warnings btw:

ld: 0711-415 WARNING: Symbol ASCII_TO_NEED is already exported.
ld: 0711-415 WARNING: Symbol boot_DynaLoader is already exported.
ld: 0711-415 WARNING: Symbol NATIVE_TO_NEED is already exported.
ld: 0711-415 WARNING: Symbol Perl__add_range_to_invlist is already exported.
ld: 0711-415 WARNING: Symbol Perl__core_swash_init is already exported.
[~1150 more]
ld: 0711-415 WARNING: Symbol PL_Yes is already exported.
ld: 0711-415 WARNING: Symbol PL_Zero is already exported.

Greetings,

Andres Freund

On 2022-09-13 00:15:55 -0700, Andres Freund wrote:
> On 2022-09-12 23:39:04 -0700, Noah Misch wrote:
> > On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:
> > > > > > -qvisibility option specifies visibility attributes for entities. Entity
> > > > > >  visibility attributes describe whether and how entities defined in one
> > > > > >  module can be referenced or used in other modules. Visibility attributes
> > > > > >  affect entities with external linkage only, and cannot increase the
> > > > > >  visibility of other entities.
> > > > >
> > > > >
> > > > > Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
> > > > > we should increase our requirement to 13.1 rather than revert back to the
> > > > > higher complexity way of building modules.
> > > >
> > > > Perhaps.  That would demolish two buildfarm members.
> > >
> > > Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
> > > available?
> >
> > I don't see 13.x, but I might not know where to look.
> 
> Looks like it's not available on gcc111, just on gcc119, there it's at
> /opt/IBM/xlc/13.1.3/bin/xlc
> 
> 
> > > > What does PostgreSQL lose if you revert fe6a64a?
> > >
> > > A somewhat readable Makefile.shlib. The prior state is complicated, specific
> > > to AIX. I've several times spent quite a bit of time understanding it - not
> > > helped by the complete lack of comments. It's not frequent enough to stay in
> > > my brain unfortunately, so I'll have to relearn it again next time. And we'll
> > > need to continue supporting Makefile.shlib for quite a while, so ...
> >
> > I'm voting for a revert, then.  Code beautification is nice when it just
> > works, but desupporting this compiler to land this particular code
> > beautification makes the net value go negative for me.  I can understand your
> > feeling differently based on your recent experiences.
> 
> IMO this more than just a beautification - the difference is functional in
> nature. I'd agree if it just were a reformating or such, but this isn't that.
> 
> I just tested, and with the mlkdexport.sh approach we end up with all symbols
> exported (-fvisibility=hidden) in extension .so's - AIX will be the only
> platform unable to restrict symbol exports. That's bound to lead to problems
> we'll only have on AIX.
> 
> The more special-case-y some platform is, the higher the cost of supporting it
> is. I'm somewhat ok with supporting niche stuff, but then the people desiring
> that support also need to do the work of making that bearable for everyone
> else.

Worth noting that xlc 12.1 appears to be EOL:

https://www.ibm.com/support/pages/unsupported-c-and-c-compilers-and-debuggers-links-latest-updates

"This page contains links to the final updates (Fix Packs, PTFs, etc) for the
non-supported IBM C and C++ compiler and debugger products. This document is
provided "AS IS" since those compilers are no longer supported.
...
XL C for AIX Fix Pack 23 (May 2021 PTF) for 12.1".

The successor, 13.1.0 is from 06 June 2014, which in turn has been superceeded
by 16.1 (16.1.0 14 December 2018) and 17.1 (17.1.0 17 September 2021).

Greetings,

Andres Freund

Hi,

On 2022-09-11 11:19:25 -0700, Andres Freund wrote:
> We could also try to fall back to using -Wl,b,expall for extension modules.

expall doesn't work, because of our use of identifiers in reserved namespaces,
e.g. _PG_init:

       expall
            Exports all global symbols, except imported symbols, unreferenced symbols defined in archive members, and
symbolsbeginning with an underscore
 
            (_). You can export additional symbols by listing them in an export file or using the expfull option. This
optiondoes not affect symbols
 
            exported by the autoexp option.

However, there also is 'expfull' - IIRC that causes problems when used for
postgres, but appears to be fine for .so's.

So we could work around the xlc 12.1 issue with something like the attached
patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
have to wait a while to see more

I think it'd be considerably better to just not support xlc < 13.1 though.


This won't fix sungazer, that just needs the perl wrapper removed...

Greetings,

Andres Freund

Hi,

On 2022-09-13 11:25:30 -0700, Andres Freund wrote:
> On 2022-09-11 11:19:25 -0700, Andres Freund wrote:
> > We could also try to fall back to using -Wl,b,expall for extension modules.
> 
> expall doesn't work, because of our use of identifiers in reserved namespaces,
> e.g. _PG_init:
> 
>        expall
>             Exports all global symbols, except imported symbols, unreferenced symbols defined in archive members, and
symbolsbeginning with an underscore
 
>             (_). You can export additional symbols by listing them in an export file or using the expfull option.
Thisoption does not affect symbols
 
>             exported by the autoexp option.
> 
> However, there also is 'expfull' - IIRC that causes problems when used for
> postgres, but appears to be fine for .so's.
> 
> So we could work around the xlc 12.1 issue with something like the attached
> patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
> have to wait a while to see more

Both passed check-world.

- Andres

Andres Freund <andres@anarazel.de> writes:
> So we could work around the xlc 12.1 issue with something like the attached
> patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
> have to wait a while to see more

Shouldn't that be more like

+  if test "$pgac_cv_prog_CC_cflags__qvisibility_hidden" != "yes"; then
+    CFLAGS_SL_MODULE='$CFLAGS_SL_MODULE -Wl,-b,expfull'
+  fi

to avoid losing whatever we found out before that?

> I think it'd be considerably better to just not support xlc < 13.1 though.

A three-line patch doesn't seem like an unreasonable thing to carry,
at least till these systems go out of support.  We've jumped through
much higher hoops in the past to support niche platforms.

            regards, tom lane

Hi,

On 2022-09-13 22:45:46 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > So we could work around the xlc 12.1 issue with something like the attached
> > patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
> > have to wait a while to see more
>
> Shouldn't that be more like
>
> +  if test "$pgac_cv_prog_CC_cflags__qvisibility_hidden" != "yes"; then
> +    CFLAGS_SL_MODULE='$CFLAGS_SL_MODULE -Wl,-b,expfull'
> +  fi
>
> to avoid losing whatever we found out before that?

Yes, you're right. There's nothing today, but we shouldn't assume that.


> > I think it'd be considerably better to just not support xlc < 13.1 though.
>
> A three-line patch doesn't seem like an unreasonable thing to carry,
> at least till these systems go out of support.  We've jumped through
> much higher hoops in the past to support niche platforms.

xlc 12.1 *is* out of support and there's a newer compiler installed.

The amount of lines itself doesn't bother me. What does is that it makes an
already complicated platform more complicated: All symbols from extension
.so's are exported, but only for old and unsupported versions of the
proprietary compiler.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2022-09-13 22:45:46 -0400, Tom Lane wrote:
>> A three-line patch doesn't seem like an unreasonable thing to carry,
>> at least till these systems go out of support.  We've jumped through
>> much higher hoops in the past to support niche platforms.

> xlc 12.1 *is* out of support and there's a newer compiler installed.

AIX 7.1 will go EOL in May.  It seems to me to be reasonable to keep
these animals running till then and then shut them down [1].  In the
meantime, messing with their configurations seems like an unreasonable
amount of work compared to applying this patch.

> The amount of lines itself doesn't bother me. What does is that it makes an
> already complicated platform more complicated: All symbols from extension
> .so's are exported, but only for old and unsupported versions of the
> proprietary compiler.

I have no objection to not supporting the case in the meson scripts,
if that'd make you any happier.

            regards, tom lane

[1] Unless the gcc compile farm shuts the machine down sooner,
which is entirely possible.

Hi,

On 2022-09-15 16:48:04 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-09-13 22:45:46 -0400, Tom Lane wrote:
> >> A three-line patch doesn't seem like an unreasonable thing to carry,
> >> at least till these systems go out of support.  We've jumped through
> >> much higher hoops in the past to support niche platforms.
> 
> > xlc 12.1 *is* out of support and there's a newer compiler installed.
> 
> AIX 7.1 will go EOL in May.  It seems to me to be reasonable to keep
> these animals running till then and then shut them down [1].  In the
> meantime, messing with their configurations seems like an unreasonable
> amount of work compared to applying this patch.

I guess I think updating the path to the compiler in the config isn't all that
heavy a lift, compared to making others understand this stuff, but...  I'll
push a revised patch.


> > The amount of lines itself doesn't bother me. What does is that it makes an
> > already complicated platform more complicated: All symbols from extension
> > .so's are exported, but only for old and unsupported versions of the
> > proprietary compiler.
> 
> I have no objection to not supporting the case in the meson scripts,
> if that'd make you any happier.

Meson doesn't, so far, support xlc, just gcc on AIX. It'd probably not too
hard to teach it, but I'm not planning to do so - doing the work to make aix
w/ gcc work was already plenty effort on my end.

Greetings,

Andres Freund