Обсуждение: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

Поиск
Список
Период
Сортировка

I am currently upgrading a Postgres cluster running on a Centos machine from version 9.6.22 to 12.7. Post the install of 12.7 binaries, I am running the pg_upgrade command:

/usr/pgsql-12/bin/pg_upgrade -b /usr/pgsql-9.6/bin/ -B /usr/pgsql-12/bin/ -d /<path>/9.6/data.old -D /<path>/12/data/ -j 12 -r -v -k

But, I am getting multiple errors during the "Restoring database schemas in the new cluster" step:

pg_restore: creating TABLE "schema1.table1"
pg_restore: creating TABLE "schema1.table2"
pg_restore: creating TABLE "schema1.table3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 358; 1259 793395 TABLE table3 postgres
pg_restore: error: could not execute query: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly    This probably means the server terminated abnormally    before or while processing the request.
Command was:
-- For binary upgrade, must preserve pg_type oid
SELECT pg_catalog.binary_upgrade_set_next_pg_type_oid('793397'::pg_catalog.oid);


-- For binary upgrade, must preserve pg_type array oid
SELECT pg_catalog.binary_upgrade_set_next_array_pg_type_oid('793396'::pg_catalog.oid);


-- For binary upgrade, must preserve pg_class oids
SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('793395'::pg_catalog.oid);

CREATE TABLE "schema1"."table3" (    "date_key" integer NOT NULL,    "col1" character varying(32),    "col2" character varying(32),    "col3" character varying(32) NOT NULL
);

-- For binary upgrade, set heap's relfrozenxid and relminmxid
UPDATE pg_catalog.pg_class
SET relfrozenxid = '1491955518', relminmxid = '1'
WHERE oid = '"schema1"."table3"'::pg_catalog.regclass;

Another log file:

pg_restore: creating TABLE "schema99.table1"
pg_restore: creating SEQUENCE "schema99.sequence1"
pg_restore: creating SEQUENCE OWNED BY "schema99.sequence1"
pg_restore: creating TABLE "schema99.table2"
pg_restore: creating SEQUENCE "schema99.sequence2"
pg_restore: creating SEQUENCE OWNED BY "schema99.sequence2"
pg_restore: creating SEQUENCE "schema99.sequence2"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 347; 1259 1949074 SEQUENCE sequence2 postgres
pg_restore: error: could not execute query: ERROR:  relation "sequence2" already exists
Command was:
-- For binary upgrade, must preserve pg_class oids
SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('1949074'::pg_catalog.oid);


-- For binary upgrade, must preserve pg_type oid
SELECT pg_catalog.binary_upgrade_set_next_pg_type_oid('1949075'::pg_catalog.oid);

CREATE SEQUENCE "schema99"."sequence2"    START WITH 1    INCREMENT BY 1    NO MINVALUE    NO MAXVALUE    CACHE 1;

Restore command from the logs:

"/usr/pgsql-12/bin/pg_restore" --host /var/lib/pgsql --port 50432 --username postgres --create --exit-on-error --verbose --dbname template1 "pg_upgrade_dump_16405.custom" >> "pg_upgrade_dump_16405.log" 2>&1

Similarly, at each run a different dump file fails (pg_upgrade_dump_16405.custom/pg_upgrade_dump_16404.custom). Has anyone faced similar issues during upgrade to Postgres 12.7?


There isn't much in the pg_upgrade_server.log file. It looks like the server starts and stops for the old and new data files. pg_upgrade_server.log file - textuploader.com/ta9o3 


Under /<path>/12/data/log. I see these lines:

```
LOG: background worker "logical replication launcher" (PID 35687) exited with exit code 1
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down at 2021-07-30 01:33:51 GMT
LOG: database system is ready to accept connections ERROR: relation "sequence2" already exists
```


I am getting an error even after only initDB and running upgrade. Any suggestions on what could be going wrong?

Thanks in advance!

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
Vijaykumar Jain
Дата:


On Sat, Jul 31, 2021, 4:00 AM Dhanush D <dhanushdk94@gmail.com> wrote:

I am currently upgrading a Postgres cluster running on a Centos machine from version 9.6.22 to 12.7. Post the install of 12.7 binaries, I am running the pg_upgrade command:

/usr/pgsql-12/bin/pg_upgrade -b /usr/pgsql-9.6/bin/ -B /usr/pgsql-12/bin/ -d /<path>/9.6/data.old -D /<path>/12/data/ -j 12 -r -v -k

But, I am getting multiple errors during the "Restoring database schemas in the new cluster" step:

pg_restore: creating TABLE "schema1.table1"
pg_restore: creating TABLE "schema1.table2"
pg_restore: creating TABLE "schema1.table3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 358; 1259 793395 TABLE table3 postgres
pg_restore: error: could not execute query: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly    This probably means the server terminated abnormally    before or while processing the request.
Command was:
-- For binary upgrade, must preserve pg_type oid
SELECT pg_catalog.binary_upgrade_set_next_pg_type_oid('793397'::pg_catalog.oid);


-- For binary upgrade, must preserve pg_type array oid
SELECT pg_catalog.binary_upgrade_set_next_array_pg_type_oid('793396'::pg_catalog.oid);


-- For binary upgrade, must preserve pg_class oids
SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('793395'::pg_catalog.oid);

CREATE TABLE "schema1"."table3" (    "date_key" integer NOT NULL,    "col1" character varying(32),    "col2" character varying(32),    "col3" character varying(32) NOT NULL
);

-- For binary upgrade, set 


```
LOG: background worker "logical replication launcher" (PID 35687) exited with exit code 1
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down at 2021-07-30 01:33:51 GMT
LOG: database system is ready to accept connections ERROR: relation "sequence2" already exists
```


I am getting an error even after only initDB and running upgrade. Any suggestions on what could be going wrong?

Thanks in advance!


Just asking, is it possible for you to a schema dump from old db and restore on the newer db, does it throw any errors.
If the db is not really big, maybe dump db with data and restore it on the newer db server ( use the 12.x postgresql cli for dump restore)

If both these go through fine, then it would be easy to take any unrelated  ddl, data issues out to focus on pg_upgrade.

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
"David G. Johnston"
Дата:
On Friday, July 30, 2021, Dhanush D <dhanushdk94@gmail.com> wrote:

pg_restore: error: could not execute query: FATAL:  terminating connection due to administrator command

Did you control-C or something, maybe remote connection dropped?

 
pg_restore: creating SEQUENCE "schema99.sequence2"
pg_restore: creating SEQUENCE OWNED BY "schema99.sequence2"
pg_restore: creating SEQUENCE "schema99.sequence2"
Is your source database corrupted such that this duplication appears in its catalog?

David J.

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
Dhanush D
Дата:
Thanks for your response, David. 

I realized I was running multiple threads to ("-j 12") run the pg_upgrade. If one thread fails, it sends a kill to other threads. Hence the fatal error.

However, running it as a single thread always fails at the below step:
 
Screen Shot 2021-08-02 at 11.38.20 AM.png

The dump file has something like this: 

Screen Shot 2021-08-02 at 11.40.01 AM.png
Multiple "Create sequence" and "Drop sequence". However the drop sequence doesn't seem to drop the sequence cleanly hence failing the restore process.

Are you or anyone aware of this issue while running pg_upgrade from 9.6 to 12.7?

On Fri, 30 Jul 2021 at 23:07, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Friday, July 30, 2021, Dhanush D <dhanushdk94@gmail.com> wrote:

pg_restore: error: could not execute query: FATAL:  terminating connection due to administrator command

Did you control-C or something, maybe remote connection dropped?

 
pg_restore: creating SEQUENCE "schema99.sequence2"
pg_restore: creating SEQUENCE OWNED BY "schema99.sequence2"
pg_restore: creating SEQUENCE "schema99.sequence2"
Is your source database corrupted such that this duplication appears in its catalog?

David J.

Вложения

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
"David G. Johnston"
Дата:
On Mon, Aug 2, 2021 at 11:42 AM Dhanush D <dhanushdk94@gmail.com> wrote:
Multiple "Create sequence" and "Drop sequence". However the drop sequence doesn't seem to drop the sequence cleanly hence failing the restore process.

It shouldn't be dropping at all - it assumes it is starting from an empty slate.

Same question as before, on the 9.6 instance, does this sequence actually show up twice?  If so, how it came to be that way might be interesting, and others may wish to investigate, but ultimately one of the two would have to be removed (not something I feel qualified to walk someone else through via email) from the 9.6, after which the upgrade should get past that point.

David J.

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
Dhanush D
Дата:
It doesn't show up twice based on these checks (Is there any other way I can check if the seq is showing up twice?).
 
Screen Shot 2021-08-02 at 12.12.06 PM.png

I'll need to drop the table completely since the column is dependent on the seq, to fix forward. Wonder how/why the duplicates of create sequences.


On Mon, 2 Aug 2021 at 12:06, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Mon, Aug 2, 2021 at 11:42 AM Dhanush D <dhanushdk94@gmail.com> wrote:
Multiple "Create sequence" and "Drop sequence". However the drop sequence doesn't seem to drop the sequence cleanly hence failing the restore process.

It shouldn't be dropping at all - it assumes it is starting from an empty slate.

Same question as before, on the 9.6 instance, does this sequence actually show up twice?  If so, how it came to be that way might be interesting, and others may wish to investigate, but ultimately one of the two would have to be removed (not something I feel qualified to walk someone else through via email) from the 9.6, after which the upgrade should get past that point.

David J.

Вложения

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
Dhanush D
Дата:
Sharing more context on this: if I run 

/usr/pgsql-9.6/bin/pg_dump -d <db_name> -t public.auth_permission;

I see multiple "CREATE SEQUENCE public.auth_permission_id_seq"  commands. This seems unusual. 

Screen Shot 2021-08-02 at 12.41.17 PM.png

On Mon, 2 Aug 2021 at 12:19, Dhanush D <dhanushdk94@gmail.com> wrote:
It doesn't show up twice based on these checks (Is there any other way I can check if the seq is showing up twice?).
 
Screen Shot 2021-08-02 at 12.12.06 PM.png

I'll need to drop the table completely since the column is dependent on the seq, to fix forward. Wonder how/why the duplicates of create sequences.


On Mon, 2 Aug 2021 at 12:06, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Mon, Aug 2, 2021 at 11:42 AM Dhanush D <dhanushdk94@gmail.com> wrote:
Multiple "Create sequence" and "Drop sequence". However the drop sequence doesn't seem to drop the sequence cleanly hence failing the restore process.

It shouldn't be dropping at all - it assumes it is starting from an empty slate.

Same question as before, on the 9.6 instance, does this sequence actually show up twice?  If so, how it came to be that way might be interesting, and others may wish to investigate, but ultimately one of the two would have to be removed (not something I feel qualified to walk someone else through via email) from the 9.6, after which the upgrade should get past that point.

David J.

Вложения

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
"David G. Johnston"
Дата:
On Mon, Aug 2, 2021 at 12:20 PM Dhanush D <dhanushdk94@gmail.com> wrote:
It doesn't show up twice based on these checks (Is there any other way I can check if the seq is showing up twice?).


 SELECT * FROM pg_class WHERE relkind = 'S' AND relname ~ 'permission';

I made the "relname" a bit broad, you should see the exact name (add an order by...).  relkind=S are sequences.

David J.

Re: Postgres 9.6 to 12.7 pg_upgrade error - terminating connection due to administrator command

От
Luca Ferrari
Дата:
On Mon, Aug 2, 2021 at 10:31 PM Dhanush D <dhanushdk94@gmail.com> wrote:
>
> Sharing more context on this: if I run
>
> /usr/pgsql-9.6/bin/pg_dump -d <db_name> -t public.auth_permission;
>
> I see multiple "CREATE SEQUENCE public.auth_permission_id_seq"  commands. This seems unusual.
>

Seems there is either a problem on the catalogs (as suggested by
David) or there is something within your template databases (but I
would not expect it on a clean installation).
Doing a catalog reindex could be a good try.
Also I would not do a pg_upgrade thru three major releases, I would
spend some time doing a more conservative approach.

Luca