Re: [PATCH] Add GitLab CI to PostgreSQL

Hi,

On 2023-07-04 23:44:45 +0000, Newhouse, Robin wrote:
> I propose the attached patch to be applied on the 'master' branch
> of PostgreSQL to add GitLab CI automation alongside Cirrus CI in the PostgreSQL repository.
> 
> It is not intended to be a replacement for Cirrus CI, but simply suggestion for the
> PostgreSQL project to host centrally a Gitlab CI definition for those who prefer to use
> it while developing/testing PostgreSQL.

One way to avoid duplicated CI definition could be to use for gitlab-ci to use
cirrus-cli to run the cirrus CI tests within gitlab ci.

Realistically I think adding a separate CI definition would entail committers
needing to run that CI at least occasionally. If we make the different CI envs
more similar, that becomes less of a necessity.


> +default:
> +  # Base image for builds and tests unless otherwise defined
> +  image: fedora:latest
> +  # Extend build jobs to have longer timeout as the default GitLab
> +  # timeout (1h) is often not enough
> +  timeout: 3h

IMO we shouldn't add CI that doesn't complete within well under an hour, it's
too expensive workflow wise.



> +fedora:
> +  stage: build
> +  variables:
> +    GIT_STRATEGY: fetch
> +    GIT_SUBMODULE_STRATEGY: normal
> +  script:
> +    # Install dependencies
> +    - yum install -y yum-utils perl
> +    - yum-builddep -y postgresql
> +    - *build-postgres-def

My experience is that installing dependencies on each run is way too slow to
be practical. I also found that it often causes temporary failures due to
network issues etc.  For cirrus-ci we create VM and docker images on a regular
schedule (three times a week right now) - if there's interest in building
fedora containers that'd be easy.

I'd be open to switching one of the cirrus-CI tasks over to fedora, fwiw.


> +# From https://github.com/postgres/postgres/blob/master/.cirrus.yml
> +.create-user: &create-user-def
> +    - useradd -m postgres
> +    - chown -R postgres:postgres .
> +    - mkdir -p ${CCACHE_DIR}
> +    - chown -R postgres:postgres ${CCACHE_DIR}
> +    - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf
> +    - su postgres -c "ulimit -l -H && ulimit -l -S"
> +    # Can't change container's kernel.core_pattern. Postgres user can't write
> +    # to / normally. Change that.
> +    - chown root:postgres /
> +    - chmod g+rwx /

If we need duplicated stanzas like this, we should instead move them out into
scripts that we can use from different CI environments.


> +# Similar to https://github.com/postgres/postgres/blob/master/.cirrus.yml
> +fedora meson:
> +  stage: build
> +  variables:
> +    GIT_STRATEGY: fetch
> +    GIT_SUBMODULE_STRATEGY: normal
> +  script:
> +    # Meson system only exists on master branch currently

Master and 16 now...


> +    - if [ ! -f meson.build ]; then exit 0; fi
> +    # Install dependencies
> +    - yum install -y yum-utils perl perl-IPC-Run meson ninja-build
> +    - yum-builddep -y postgresql
> +    # Create postgres user
> +    - *create-user-def
> +    # Configure
> +    - su postgres -c 'meson setup --buildtype=debug --auto-features=disabled -Dtap_tests=enabled build'
> +    # Build
> +    - su postgres -c 'ninja -C build -j 2'
> +    # Minimal test
> +    - su postgres -c 'meson test $MTEST_ARGS --num-processes 2 tmp_install cube/regress pg_ctl/001_start_stop'
> +    # Run all tests
> +    - su postgres -c 'meson test $MTEST_ARGS --num-processes 2'
> +  artifacts:
> +    when: always  # Must be able to see logs
> +    paths:
> +      - build/meson-logs/testlog.txt

FWIW, that's not enough to be able to debug problems. You really also need the
log files created by failing tests.

Greetings,

Andres Freund