Обсуждение: Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64

​Hi Christoph,

This is a follow-up email based on recommendations from Heikki Linnakangas:

https://www.postgresql.org/message-id/5f048a5a-c471-50ae-30ad-385dcafc616b%40iki.fi

and Tom Lane:

https://www.postgresql.org/message-id/888692.1601489331%40sss.pgh.pa.us

> Maybe you should speak to the distribution vendors or the folk packaging PostgreSQL for those distributions, instead.

Following the initial post by Tsahi Zidenberg:

https://www.postgresql.org/message-id/099F69EE-51D3-4214-934A-1F28C0A1A7A7%40amazon.com

> The patch was tested to improve pgbench simple-update by 10% and sysbench write-only by 3% on a 64-core armv8.2 machine (AWS m6g.16xlarge).

​

​The change is to compile PostgreSQL for arm64 with an extra flag -moutline-atomics when the compiler is gcc-8.5 or gcc-9.4.

-moutline-atomics is enabled by default in all newer compilers gcc-{10, 11, trunk}, and on clang-{12, 13, trunk}.

The change is needed only for a limited number of Debian and Ubuntu releases compiled by gcc versions with the outline-atomics flag disabled by default.

Re: Pop, Sebastian
> ?Hi Christoph,
> 
> 
> This is a follow-up email based on recommendations from Heikki Linnakangas:
> 
> https://www.postgresql.org/message-id/5f048a5a-c471-50ae-30ad-385dcafc616b%40iki.fi
> 
> and Tom Lane:
> 
> https://www.postgresql.org/message-id/888692.1601489331%40sss.pgh.pa.us

Well I wouldn't exactly call these messages "recommendations".

> ?The change is to compile PostgreSQL for arm64 with an extra flag -moutline-atomics when the compiler is gcc-8.5 or
gcc-9.4.
> 
> -moutline-atomics is enabled by default in all newer compilers gcc-{10, 11, trunk}, and on clang-{12, 13, trunk}.
> 
> The change is needed only for a limited number of Debian and Ubuntu releases compiled by gcc versions with the
outline-atomicsflag disabled by default.
 

None of the Debian releases are using 8.5 or 9.4, and on the Ubuntu
side focal is using 9.3. Is that gcc version even affected?

Christoph

> on the Ubuntu side focal is using 9.3. Is that gcc version even affected?

Yes, we have backported the outline-atomics flag to Ubuntu 20.04 gcc-9.3.

On arm64 you can check whether a compiler has the option with:
$ gcc -v --help -Q 2>/dev/null | grep outline-atomics
  -moutline-atomics                   [disabled]
$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
[...]

Sebastian

Re: Pop, Sebastian
> > on the Ubuntu side focal is using 9.3. Is that gcc version even affected?
> 
> Yes, we have backported the outline-atomics flag to Ubuntu 20.04 gcc-9.3.
> 
> On arm64 you can check whether a compiler has the option with:
> $ gcc -v --help -Q 2>/dev/null | grep outline-atomics
>   -moutline-atomics                   [disabled]
> $ gcc --version
> gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

Not here:

$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
$ gcc -v --help -Q 2>/dev/null | grep outline-atomics

Did you rebuild the package locally without changing the Debian
version?

Christoph

> Not here:
> $ gcc --version
> gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> $ gcc -v --help -Q 2>/dev/null | grep outline-atomics

This needs to be on an arm64 machine.
The outline-atomics flag does not exist when targeting x86_64.
The flag has been back-ported to Ubuntu 20.04 a year and a half ago: IIRC, it went in just before the release in April
2020.

> Did you rebuild the package locally without changing the Debian version?

I have not re-compiled gcc.  The gcc-9.3 is the one distributed as part of Ubuntu 20.04.

If you need access to Graviton2 instances, we have a system of credits that we can use to fund compute time for
open-sourceCI projects. 
Let me know if you are interested in getting those compute credits, and I will take the request to my management.

Sebastian

Hi Christoph,

Could you please let me know what is the process to enable -moutline-atomics for arm64 binaries distributed by
PostgreSQLfor Ubuntu 20.04? 
Using the right LSE atomic instructions is very important for the performance of PostgreSQL on all new arm64
processors.
Please let me know how I can help to enable the flag, test, and benchmark the new packages.

Thanks,
Sebastian

Re: Pop, Sebastian
> > Not here:
> > $ gcc --version
> > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > $ gcc -v --help -Q 2>/dev/null | grep outline-atomics
> 
> This needs to be on an arm64 machine.

Oh, sure. Sorry I was missing a lot of context that wasn't immediately
clear.

> The outline-atomics flag does not exist when targeting x86_64.
> The flag has been back-ported to Ubuntu 20.04 a year and a half ago: IIRC, it went in just before the release in
April2020.
 

It is indeed present there, but not enabled:

$ for chroot in *; do echo $chroot:; schroot -c $chroot -- gcc -v --help -Q 2>&1 | grep outline-atomics; done
bionic-arm64:
bookworm-arm64:
  -moutline-atomics                   [enabled]
bullseye-arm64:
  -moutline-atomics                   [enabled]
buster-arm64:
focal-arm64:
  -moutline-atomics                   [disabled]
sid-arm64:
  -moutline-atomics                   [enabled]

The (newer) Debian dists have it enabled by default.


> > Did you rebuild the package locally without changing the Debian version?
> 
> I have not re-compiled gcc.  The gcc-9.3 is the one distributed as part of Ubuntu 20.04.

Again, that was a lot of context I didn't have.

> If you need access to Graviton2 instances, we have a system of credits that we can use to fund compute time for
open-sourceCI projects.
 
> Let me know if you are interested in getting those compute credits, and I will take the request to my management.

Thanks for the offer - we already have a arm64 build node sponsored by
Huawei that does the job for now.


Re: Pop, Sebastian
> Could you please let me know what is the process to enable -moutline-atomics for arm64 binaries distributed by
PostgreSQLfor Ubuntu 20.04?
 
> Using the right LSE atomic instructions is very important for the performance of PostgreSQL on all new arm64
processors.
> Please let me know how I can help to enable the flag, test, and benchmark the new packages.

First, we need to be convinced it's a good idea. From that I got from
the messages you were linking to, Heikki "definitely do[es]n't want to
override that decision", and Tom doesn't "find this argument terribly
convincing".

The benchmarks in
https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com
don't look like the option is a game-changer.

Even if we are convinced, we don't have the infrastructure in the
package yet to do architecture- and distribution-specific
optimizations. Would you (Amazon?) be willing to sponsor the required
package engineering?

Christoph

> The benchmarks in
> https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com
> don't look like the option is a game-changer.

Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions.
I will get the new data and share.

> Even if we are convinced, we don't have the infrastructure in the
> package yet to do architecture- and distribution-specific
> optimizations. Would you (Amazon?) be willing to sponsor the required
> package engineering?

Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04.
Please point me to the (git?) repo and the files that need to be changed.

Thanks,
Sebastian

Re: Pop, Sebastian
> > The benchmarks in
> > https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com
> > don't look like the option is a game-changer.
> 
> Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions.
> I will get the new data and share.
> 
> > Even if we are convinced, we don't have the infrastructure in the
> > package yet to do architecture- and distribution-specific
> > optimizations. Would you (Amazon?) be willing to sponsor the required
> > package engineering?
> 
> Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04.
> Please point me to the (git?) repo and the files that need to be changed.

https://salsa.debian.org/postgresql/postgresql-common.git
https://salsa.debian.org/postgresql/postgresql.git
https://git.postgresql.org/gitweb/?p=pgapt.git;a=summary

Christoph

On 11/11/21, 11:14 AM, "Christoph Berg" <myon@debian.org> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you
canconfirm the sender and know the content is safe.
 



    Re: Pop, Sebastian
    > > The benchmarks in
    > > https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com
    > > don't look like the option is a game-changer.
    >
    > Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions.
    > I will get the new data and share.
    >
    > > Even if we are convinced, we don't have the infrastructure in the
    > > package yet to do architecture- and distribution-specific
    > > optimizations. Would you (Amazon?) be willing to sponsor the required
    > > package engineering?
    >
    > Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04.
    > Please point me to the (git?) repo and the files that need to be changed.

The binary compiled with -moutline-atomics works for all arm64 architecture versions. I don't believe there would be
newpackage engineering required unless I'm missing something, but if there is, yes, we're absolutely happy to help. 
 

Thanks,
Ali

Thanks Christoph for the pointers.

I see that the source code of PostgreSQL is patched before builds:
i.e., the configure.in file has this patch applied to it
https://salsa.debian.org/postgresql/postgresql/-/blob/13/debian/patches/autoconf2.69

Following the original patch from Tsahi Zidenberg,
https://www.postgresql.org/message-id/attachment/111983/0001-Support-outline-atomics-on-aarch64.patch
I would like to add the attached patch to the list of patches applied before build.

Christoph, could you please let me know how I should test the patch?

Thanks,
Sebastian

Re: Pop, Sebastian
> Thanks Christoph for the pointers.
> 
> I see that the source code of PostgreSQL is patched before builds:
> i.e., the configure.in file has this patch applied to it
https://salsa.debian.org/postgresql/postgresql/-/blob/13/debian/patches/autoconf2.69
> 
> Following the original patch from Tsahi Zidenberg,
> https://www.postgresql.org/message-id/attachment/111983/0001-Support-outline-atomics-on-aarch64.patch
> I would like to add the attached patch to the list of patches applied before build.

That will fail on bionic.

> Christoph, could you please let me know how I should test the patch?

Apply it, and run "make check-world" on focal. The others should be
unaffected since the flag is on by default.

Christoph

>> The benchmarks in
>> https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com
>> don't look like the option is a game-changer.
>
> Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions.
> I will get the new data and share.

On an m6gd.metal with PostgreSQL v11
test: pgbench - select-only
Measure tps (transactions/s), higher is better.

build without LSE: 132143 tps
with LSE atomics: 844552 tps

That is a 6.39x speedup.

Sebastian

> That will fail on bionic.

The autoconf code does a check that the compiler supports the -moutline-atomics flag.
The change will still be fine on bionic and all other distros with a gcc without the flag.

> > Christoph, could you please let me know how I should test the patch?
> Apply it, and run "make check-world" on focal. The others should be
> unaffected since the flag is on by default.

Thanks for the instructions.  I will test the patch and let you know.

Sebastian

> Apply it, and run "make check-world" on focal. The others should be
> unaffected since the flag is on by default.

Here are the steps in my testing on focal - ubuntu 20.04:

git clone https://github.com/postgres/postgres
cd postgres
patch -p1 < a.diff
autoconf
make
make check-world

All checks passed on a c6g Graviton2 instance.

configure.in was renamed configure.ac in July 2020.
For older postgresql versions we need to patch configure.in.
See attached the patch a.diff that I used on trunk postgresql.

Thanks,
Sebastian

sudo apt install build-essential libreadline-dev zlib1g-dev bison flex
> git clone https://github.com/postgres/postgres
> cd postgres
> patch -p1 < a.diff
> autoconf
./configure
> make
> make check-world
>
> All checks passed on a c6g Graviton2 instance.

I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the
-moutline-atomicsflag. 
All checks pass on bionic on a c6g instance.
Thanks Ali for recommending the extra test.

Sebastian

Hi Christoph,

a quick follow-up email to make sure you have everything you need to enable -moutline-atomics flag on Ubuntu 20.04
buildsof PostgreSQL. 
Please let me know if you need more experiments or tests with the patch.

Thanks,
Sebastian

Re: Pop, Sebastian
> I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the
-moutline-atomicsflag.
 
> All checks pass on bionic on a c6g instance.
> Thanks Ali for recommending the extra test.

Thanks for verifying that part as well, I'll try throwing in the patch
now.

Christoph

Re: To Pop, Sebastian
> > I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the
-moutline-atomicsflag.
 
> > All checks pass on bionic on a c6g instance.
> > Thanks Ali for recommending the extra test.
> 
> Thanks for verifying that part as well, I'll try throwing in the patch
> now.

Things are looking good on PG14, but PG10 and 9.6 fail:

configure.in:485: error: possibly undefined macro: PGAC_PROG_CXX_CFLAGS_OPT

https://pgdgbuild.dus.dg-i.net/job/postgresql-10-binaries/architecture=ppc64el,distribution=focal/76/console

So we'll have it on PG 11 .. 14, unless the regression tests act up.

Christoph

> Things are looking good on PG14, but PG10 and 9.6 fail:
> configure.in:485: error: possibly undefined macro: PGAC_PROG_CXX_CFLAGS_OPT

Thanks Christoph for the heads up.
I will investigate the autoconf issue and send you a patch specific to PG10 and 9.
Most likely the PGAC_PROG_CXX_CFLAGS_OPT macro has been introduced after these releases.

Sebastian

> I will investigate the autoconf issue and send you a patch specific to PG10 and 9.
> Most likely the PGAC_PROG_CXX_CFLAGS_OPT macro has been introduced after these releases.

The above macro has been added in March 2018:
https://github.com/postgres/postgres/commit/6869b4f2584787d9e4cefaab8a4bae1ecbe63766
the patch also adds CXXFLAGS, and so before this patch there is no need to detect CXXFLAGS.

Please see attached a patch tested on Ubuntu 20.04 on the following branches:
remotes/origin/REL9_6_STABLE
remotes/origin/REL_10_STABLE

I checked that REL_11_STABLE has the CXXFLAGS and the above macro.

Thanks,
Sebastian

Thanks Christoph for the quick turn-around.

I see that builds with -moutline-atomics succeeded for Focal arm64 builds for all the branches:
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-9.6-binaries/architecture=arm64,distribution=focal/88/
https://pgdgbuild.dus.dg-i.net/job/postgresql-10-binaries/77/architecture=arm64,distribution=focal/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-11-binaries/architecture=arm64,distribution=focal/686/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-12-binaries/architecture=arm64,distribution=focal/875/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-13-binaries/architecture=arm64,distribution=focal/858/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-14-binaries/architecture=arm64,distribution=focal/651/

I see that the patch is still missing on branch 15 that is the development branch:
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-15-binaries/architecture=amd64,distribution=focal/345/
Maybe we need to add the patch to that branch as well?

How are the above binaries distributed to users?
I see for instance that pgsql 14 binaries have last been updated in 30-Sep-2021:
https://apt.postgresql.org/pub/repos/apt/dists/focal-pgdg/14/binary-arm64/
Is there another step needed to make those binaries available through the PostgreSQL Apt Repository (
https://www.postgresql.org/download/linux/ubuntu/) 

Does Canonical rely on these builds to update the Focal PostgreSQL packages?

Thanks,
Sebastian

> The packages should be accessible via the 'focal-pgdg-testing' distribution, where they initially land before eventual promotion to the 'focal-pgdg' distribution.

Hi,

Re: Pop, Sebastian
> I see that the patch is still missing on branch 15 that is the development branch:
> https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-15-binaries/architecture=amd64,distribution=focal/345/

That's amd64.

https://salsa.debian.org/postgresql/postgresql/-/blob/15/debian/patches/focal-arm64-outline-atomics

> Does Canonical rely on these builds to update the Focal PostgreSQL packages?

I don't know what Ubuntu's policy on updating released packages is.

Christoph

> That's amd64.
> https://salsa.debian.org/postgresql/postgresql/-/blob/15/debian/patches/focal-arm64-outline-atomics

Sorry about that typo.  Arm64 branch 15 looks good.  Thanks Christoph.

> I don't know what Ubuntu's policy on updating released packages is.

I took the issue to Canonical and I will follow up with them to update the PostgreSQL packages.

Thanks,
Sebastian