Trouble starting Postgresql after an upgrade

I’m having a little trouble and I hope someone can point me towards a fix. J

We’ve been using Pg 9.3.4 and just upgraded to 9.5.1. I’ve installed it on 1 server and it works. J However, when I copied those binaries over to a 2^nd server, I find that they won’t start normally. I can do “/etc/init.d/postgresql-9.5 start” from anywhere and it fails to start. Looking in the script, I believe the important line (expanded via bash -x) is:

+ echo -n 'Starting postgresql-9.5 service: '

Starting postgresql-9.5 service: + runuser -l postgres -c '/opt/pgsql-9.5/bin/postmaster -p '\''5432'\'' -D '\''/nms-db/9.5/data'\''  &'

In the pgstartup.log, I see:

LOG:  skipping missing configuration file "/home/postgres/postgresql.auto.conf"

2016-07-12 22:16:41 GMT [6066]: [2-1] user=,db=,app= FATAL:  "/home/postgres" is not a valid data directory

2016-07-12 22:16:41 GMT [6066]: [3-1] user=,db=,app= DETAIL:  File "/home/postgres/PG_VERSION" is missing.

Duh, there is no postgresql.auto.conf in the ~postgres, it's in the data dir "/nms-db/9.5/data", as is "PG_VERSION". I also don’t see anything in the main log (in /logs/pg_log/post*”) that gives me any more info.

Since it all works on the original server, I must have done something wrong on install, but I can’t figure out what. Can someone see what I did wrong from the above or a pointer to a place to look with better error messages would also be helpful.

To make the problem more interesting, I can bring the DB engine up if I use pg_ctl … but only if I’m in the data dir.

[root@nms59c ~]# pwd

/root

[root@nms59c ~]# sudo -u postgres "/opt/pgsql-9.5/bin/pg_ctl" -w -D "/nms-db/9.5/data" -o ""  start

could not change directory to "/root": Permission denied

waiting for server to start....LOG:  skipping missing configuration file "/opt/pgsql-9.5/bin/postgresql.auto.conf"

2016-07-12 22:41:30 GMT [5567]: [2-1] user=,db=,app= FATAL:  data directory "/opt/pgsql-9.5/bin" has wrong ownership

2016-07-12 22:41:30 GMT [5567]: [3-1] user=,db=,app= HINT:  The server must be started by the user that owns the data directory.

stopped waiting

pg_ctl: could not start server

Examine the log output.

[root@nms59c ~]#

[root@nms59c ~]#

[root@nms59c ~]# cd $PGDATA

[root@nms59c data]# pwd

/nms-db/9.5/data

[root@nms59c data]# sudo -u postgres "/opt/pgsql-9.5/bin/pg_ctl" -w -D "/nms-db/9.5/data" -o ""  start

waiting for server to start....2016-07-12 22:41:38 GMT [5571]: [1-1] user=,db=,app= LOG:  redirecting log output to logging collector process

2016-07-12 22:41:38 GMT [5571]: [2-1] user=,db=,app= HINT:  Future log output will appear in directory "/logs/pg_log".

done

server started

[root@nms59c data]#

That almost makes sense to me as it probably wants to write some file so perms matter, but yet I don’t understand why it wouldn’t change to the data dir itself if that’s really required.

Given the above, I’m tempted to change /etc/init.d/postgresql-9.5 so that it does “cd $PGDATA” then replace the “runuser -l postgres -c…” line to be what I have that works with pg_ctl, but I’m concerned that the error is an indication that I’ve screwed up something important that will bite me later if I don’t figure it out now. This is on a test system, so no real data is endangered; but I really need to figure this out as we automate the upgrade so it can go to real data servers.

Kevin