pg_upgrade 9.6 to 12 without 9.6 binaries

We are running Postgres 9.6 on server A.

We want to upgrade it to Postgres 12 on server B.

pg_upgrade requires the old (-b) and new (-B) binary set on the same server.

We don't want to install both binary sets on the same server as we had some library conflicts in the past.

Is there still a way to use pg_upgrade without the old (-b) binary set (I am aware of pg_dump/pg_restore)?

Markus