Promoted slave tries to archive previously archived WAL file

ls -l /mnt/pgsql/archive/

-rw-rw-rw-. 1 root root 16777216 Feb 15 09:39 000000010000000000000001

-rw-rw-rw-. 1 root root 16777216 Feb 15 09:39 000000010000000000000002

-rw-rw-rw-. 1 root root      302 Feb 15 09:39 000000010000000000000002.00000028.backup

pg-hdp-node1.kitchen.local

/var/lib/pgsql/10/data/pg_wal/:

-rw-------. 1 postgres postgres 16777216 Feb 15 09:39 000000010000000000000002

-rw-------. 1 postgres postgres      302 Feb 15 09:39 000000010000000000000002.00000028.backup

-rw-------. 1 postgres postgres 16777216 Feb 15 09:44 000000010000000000000003

-rw-------. 1 postgres postgres 16777216 Feb 15 09:39 000000010000000000000004

drwx------. 2 postgres postgres       96 Feb 15 09:44 archive_status

/var/lib/pgsql/10/data/pg_wal/archive_status:

-rw-------. 1 postgres postgres 0 Feb 15 09:39 000000010000000000000002.00000028.backup.done

-rw-------. 1 postgres postgres 0 Feb 15 09:39 000000010000000000000002.done

diff from secondary pg-hdp-node2.kitchen.local on /var/lib/pgsql/10/data/pg_wal/000000010000000000000002 and /mnt/pgsql/archive/000000010000000000000002 shows binary differences but as expected no differences for diff on primary pg-hdp-node1.kitchen.local 

Failover is performed and pg-hdp-node2.kitchen.local tries and fails to archive WAL segment 000000010000000000000002 because it has been previously archived

2019-02-15 09:54:50.518 PST [780] DETAIL:  The failed archive command was: test ! -f /mnt/pgsql/archive/000000010000000000000002 && cp pg_wal/000000010000000000000002 /mnt/pgsql/archive/000000010000000000000002

Based on this thread https://www.postgresql.org/message-id/11b405a6-2176-9510-bf5b-ea9c0e860635%40pgmasters.net it is suggested to handle this case by reporting success but in my case contents are different. I would think that previously archived 000000010000000000000002 is the right WAL segment.

So my questions are as follows:

Why WAL segments differ?

How should this be resolved on the new primary?