Re: [External] Re: postgresql-11 installation errors via deb packageon ubuntu 16

Thanks Chris for responding.

ands to your queries:

this is a  fresh installation on a fresh server.
i am aware of fixes :) and it works well.

but this breaks automation, and it never used to happen earlier,
only recently.


i am not saying where the bug is, it may be something on us, but it is
difficult to pin point coz i am unable to reproduce it.
it happens occasionally.

i am tracking /var/lib/postgresql and /etc/postresql folder in a
parallel session before the package installation.


something like this in a infinite loop
find /var/lib/postgresql* | xargs ls -ld
find /etc/postgresql* | xargs ls -ld
echo "######################"

which gives

find: '/var/lib/postgresql*': No such file or directory
drwx------ 5 root root 4096 Mar 18 14:12 .
find: '/etc/postgresql*': No such file or directory
drwx------ 5 root root 4096 Mar 18 14:12 .

then later when the package installation happens,

it is
#######################
drwxr-xr-x  3 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql
drwxr-xr-x  3 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql/11
drwx------ 19 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql/11/main
drwx------  3 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/base
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/base/1
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/global
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_commit_ts
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_dynshmem
drwx------  4 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical/mappings
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical/snapshots
drwx------  4 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact/members
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact/offsets
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_notify
-rw-------  1 postgres postgres 8192 Mar 18 20:09
/var/lib/postgresql/11/main/pg_notify/0000
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_replslot
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_serial
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_snapshots
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_stat
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_stat_tmp
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_subtrans
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_tblspc
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_twophase
-rw-------  1 postgres postgres    3 Mar 18 20:09
/var/lib/postgresql/11/main/PG_VERSION
drwx------  3 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_wal
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_wal/archive_status
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_xact
-rw-------  1 postgres postgres    0 Mar 18 20:09
/var/lib/postgresql/11/main/postgresql.conf
-r--------  1 root     root       32 Mar 18 20:05
/var/lib/postgresql/.database_backup_encryption_key_file
drwxr-xr-x 3 postgres postgres 4096 Mar 18 20:09 /etc/postgresql
drwxr-xr-x 3 postgres postgres 4096 Mar 18 20:09 /etc/postgresql/11
drwxr-xr-x 2 root     root     4096 Mar 18 20:09 /etc/postgresql/11/main
drwxr-xr-x 3 root     root     4096 Mar 18 20:07 /etc/postgresql-common
-rw-r--r-- 1 root     root     1325 Mar 18 20:07
/etc/postgresql-common/createcluster.conf
drwxr-xr-x 2 root     root     4096 Jan 28 15:30
/etc/postgresql-common/pg_upgradecluster.d
-rw-r--r-- 1 root     root      607 Mar 18 20:07 /etc/postgresql-common/root.crt
-rw-r--r-- 1 root     root       90 Jan 28 15:30
/etc/postgresql-common/supported_versions
-rw-r--r-- 1 root     root     1113 Dec  9  2016
/etc/postgresql-common/user_clusters


i think the most important one being
```/var/lib/postgresql/11/main/postgresql.conf```
which is touched? and the populated and moved to /etc/postgresql/ path
and the initdb initializes the db.

on a bad installation, that file is not generated, is what i think
results in the move error?

this was the issue over last few days, i spun up new machines today
and did not get errors.
i am trying to spin up more machines, will report if it still get those errors.

on a good installation,

i see the below in the logs.


`````````
Creating new PostgreSQL cluster 11/main ...
/usr/lib/postgresql/11/bin/initdb -D /var/lib/postgresql/11/main
--auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/11/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main
-l logfile start

Ver Cluster Port Status Owner    Data directory              Log file

in a good installation:
#######################
drwxr-xr-x  3 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql
drwxr-xr-x  3 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql/11
drwx------ 19 postgres postgres 4096 Mar 18 20:09 /var/lib/postgresql/11/main
drwx------  3 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/base
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/base/1
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/global
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_commit_ts
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_dynshmem
drwx------  4 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical/mappings
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_logical/snapshots
drwx------  4 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact/members
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_multixact/offsets
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_notify
-rw-------  1 postgres postgres 8192 Mar 18 20:09
/var/lib/postgresql/11/main/pg_notify/0000
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_replslot
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_serial
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_snapshots
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_stat
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_stat_tmp
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_subtrans
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_tblspc
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_twophase
-rw-------  1 postgres postgres    3 Mar 18 20:09
/var/lib/postgresql/11/main/PG_VERSION
drwx------  3 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_wal
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_wal/archive_status
drwx------  2 postgres postgres 4096 Mar 18 20:09
/var/lib/postgresql/11/main/pg_xact
-rw-------  1 postgres postgres    0 Mar 18 20:09
/var/lib/postgresql/11/main/postgresql.conf
-r--------  1 root     root       32 Mar 18 20:05
/var/lib/postgresql/.somefile   -------this file is created via puppet
, but can be ignored if i read the pre/post install script right.
(this exists in both good and bad installations)

drwxr-xr-x 3 postgres postgres 4096 Mar 18 20:09 /etc/postgresql
drwxr-xr-x 3 postgres postgres 4096 Mar 18 20:09 /etc/postgresql/11
drwxr-xr-x 2 root     root     4096 Mar 18 20:09 /etc/postgresql/11/main
drwxr-xr-x 3 root     root     4096 Mar 18 20:07 /etc/postgresql-common
-rw-r--r-- 1 root     root     1325 Mar 18 20:07
/etc/postgresql-common/createcluster.conf
drwxr-xr-x 2 root     root     4096 Jan 28 15:30
/etc/postgresql-common/pg_upgradecluster.d
-rw-r--r-- 1 root     root      607 Mar 18 20:07 /etc/postgresql-common/root.crt
-rw-r--r-- 1 root     root       90 Jan 28 15:30
/etc/postgresql-common/supported_versions
-rw-r--r-- 1 root     root     1113 Dec  9  2016
/etc/postgresql-common/user_clusters
``````````````````


and then when it copies fine
drwxr-xr-x 3 postgres postgres  4096 Mar 18 20:09 /etc/postgresql
drwxr-xr-x 3 postgres postgres  4096 Mar 18 20:09 /etc/postgresql/11
drwxr-xr-x 3 postgres postgres  4096 Mar 18 20:09 /etc/postgresql/11/main
drwxr-xr-x 2 postgres postgres  4096 Mar 18 20:09 /etc/postgresql/11/main/conf.d
-rw-r--r-- 1 postgres postgres   315 Mar 18 20:09
/etc/postgresql/11/main/environment
-rw-r--r-- 1 postgres postgres   143 Mar 18 20:09
/etc/postgresql/11/main/pg_ctl.conf
-rw-r----- 1 postgres postgres  4686 Mar 18 20:09
/etc/postgresql/11/main/pg_hba.conf
-rw-r----- 1 postgres postgres  1636 Mar 18 20:09
/etc/postgresql/11/main/pg_ident.conf
-rw-r--r-- 1 postgres postgres 24027 Mar 18 20:09
/etc/postgresql/11/main/postgresql.conf
-rw-r--r-- 1 postgres postgres   317 Mar 18 20:09
/etc/postgresql/11/main/start.conf
drwxr-xr-x 3 root     root      4096 Mar 18 20:07 /etc/postgresql-common
-rw-r--r-- 1 root     root      1325 Mar 18 20:07
/etc/postgresql-common/createcluster.conf
drwxr-xr-x 2 root     root      4096 Jan 28 15:30
/etc/postgresql-common/pg_upgradecluster.d
-rw-r--r-- 1 root     root       607 Mar 18 20:07
/etc/postgresql-common/root.crt
-rw-r--r-- 1 root     root        90 Jan 28 15:30
/etc/postgresql-common/supported_versions
-rw-r--r-- 1 root     root      1113 Dec  9  2016
/etc/postgresql-common/user_clusters



************
in a bad installation,

the contents of the file /var/log/apt/term.log

Log started: 2019-03-18  22:11:33
Selecting previously unselected package pgdg-keyring.
(Reading database ... ^M(Reading database ... 5%^M(Reading database
... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Rea
ding database ... 25%^M(Reading database ... 30%^M(Reading database
... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Rea
ding database ... 50%^M(Reading database ... 55%^M(Reading database
... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Rea
ding database ... 75%^M(Reading database ... 80%^M(Reading database
... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Rea
ding database ... 100%^M(Reading database ... 92546 files and
directories currently installed.)
Preparing to unpack .../pgdg-keyring_2018.2_all.deb ...
Unpacking pgdg-keyring (2018.2) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_199.pgdg16.04+1_all.deb ...
Unpacking postgresql-client-common (199.pgdg16.04+1) ...
Selecting previously unselected package postgresql-client-11.
Preparing to unpack .../postgresql-client-11_11.2-1.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-client-11 (11.2-1.pgdg16.04+1) ...
Setting up pgdg-keyring (2018.2) ...
Removing apt.postgresql.org key from trusted.gpg: OK
Setting up postgresql-client-common (199.pgdg16.04+1) ...
Setting up postgresql-client-11 (11.2-1.pgdg16.04+1) ...
update-alternatives: using /usr/share/postgresql/11/man/man1/psql.1.gz
to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode


Selecting previously unselected package postgresql-common.
Preparing to unpack .../postgresql-common_199.pgdg16.04+1_all.deb ...
Adding 'diversion of /usr/bin/pg_config to
/usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (199.pgdg16.04+1) ...
Selecting previously unselected package pgbouncer.
Preparing to unpack .../pgbouncer_1.7.2-3.pgdg16.04+1_amd64.deb ...
Unpacking pgbouncer (1.7.2-3.pgdg16.04+1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.17) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libevent-2.0-5:amd64 (2.0.21-stable-2ubuntu0.16.04.1) ...
Setting up ssl-cert (1.0.37) ...
Setting up postgresql-common (199.pgdg16.04+1) ...
Adding user postgres to group ssl-cert


Creating config file /etc/postgresql-common/createcluster.conf with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up pgbouncer (1.7.2-3.pgdg16.04+1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.17) ...
Processing triggers for ureadahead (0.100.0-19) ...
Log ended: 2019-03-18  22:12:21

and then

Log started: 2019-03-18  22:36:27
Selecting previously unselected package postgresql-11.
(Reading database ... ^M(Reading database ... 5%^M(Reading database
... 10%^M(Reading database ... 15%^M(Reading database ...
20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading
database ... 35%^M(Reading database ... 40%^M(Reading database ...
45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading
database ... 60%^M(Reading database ... 65%^M(Reading database ...
70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading
database ... 85%^M(Reading database ... 90%^M(Reading database ...
95%^M(Reading database ... 100%^M(Reading database ... 100550 files
and directories currently installed.)
Preparing to unpack .../postgresql-11_11.2-1.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-11 (11.2-1.pgdg16.04+1) ...
Processing triggers for postgresql-common (199.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up postgresql-11 (11.2-1.pgdg16.04+1) ...
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
/usr/sbin/pam_getenv line 78.
Configuring already existing cluster (configuration:
/etc/postgresql/11/main, data: /var/lib/postgresql/11/main, owner:
112:118)
Error: move_conffile: required configuration file
/var/lib/postgresql/11/main/postgresql.conf does not exist
Error: could not create default cluster. Please create it manually with

  pg_createcluster 11 main --start

or a similar command (see 'man pg_createcluster').
update-alternatives: using
/usr/share/postgresql/11/man/man1/postmaster.1.gz to provide
/usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Log ended: 2019-03-18  22:36:29


now
id postgres
uid=112(postgres) gid=118(postgres) groups=118(postgres),117(ssl-cert)

it says owner is 112 and group 118  which is created by the package only.
none of our puppet code creates that user afaik.


Again, since this is not reproducible, i am not saying if this is a
bug, or a race condition or something that i am not yet aware on my
side,
my only reference being, we spin up a *new* server, that server is
provisioned via puppet.

but just asking if there are any pointers that we can look into :)

just in case, our package installation is via
https://github.com/puppetlabs/puppetlabs-postgresql/blob/master/manifests/server/install.pp
and it is a one time installation only :)

no messages in syslogs show there was postgresql preinstalled and then
this was a reinstallation etc.
I am still trying to reproduce, and will update this thread if i find
anything concrete.


Regards,
Vijay

On Tue, Mar 19, 2019 at 3:35 PM Christoph Berg <myon@debian.org> wrote:
>
> Re: Vijaykumar Jain 2019-03-18 <CAE7uO5iYDWe4rcH+rDw6NOd_HX75sUaggouL-5FjzjZGbBgqYA@mail.gmail.com>
> > Hey Guys,
> >
> > I hope this is the right dist.
>
> Hi,
>
> it is, thanks for the report.
>
> > we are having issues reported similar to
> > https://stackoverflow.com/questions/2748607/how-to-thoroughly-purge-and-reinstall-postgresql-on-ubuntu
> > which was ages back :) , but it is repeated.
>
> > Setting up postgresql-11 (11.2-1.pgdg16.04+1) ...
> > Unescaped left brace in regex is deprecated, passed through in regex;
> > marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at
> > /usr/sbin/pam_getenv line 78.
>
> (This is a bug in the pam package, not in postgresql-*.)
>
> > Configuring already existing cluster (configuration:
> > /etc/postgresql/11/main, data: /var/lib/postgresql/11/main, owner:
> > 112:118)
> > Error: move_conffile: required configuration file
> > /var/lib/postgresql/11/main/postgresql.conf does not exist
> > Error: could not create default cluster. Please create it manually with
> >
> >   pg_createcluster 11 main --start
>
> This happens when you didn't fully remove the old cluster and
> /var/lib/postgresql/11/main is still present:
>
> $ sudo pg_createcluster 11 foo
> $ sudo rm -rf /etc/postgresql/11/foo/
> $ sudo pg_createcluster 11 foo
> Configuring already existing cluster (configuration: /etc/postgresql/11/foo, data: /var/lib/postgresql/11/foo, owner:
111:118)
> Error: move_conffile: required configuration file /var/lib/postgresql/11/foo/postgresql.conf does not exist
>
> To fix, simply delete the data directory.
>
> Christoph