Issues with ON CONFLICT UPDATE and REINDEX CONCURRENTLY

Hello, hackers.

While testing my work on (1) I was struggling with addressing a strange issue with ON CONFLICT UPDATE and REINDEX CONCURRENTLY.

After some time, I have realized the same issue persists on the master branch as well :)

I have prepared two TAP tests to reproduce the issues (2), also in attachment.

First one, does the next thing:

    CREATE UNLOGGED TABLE tbl(i int primary key, updated_at timestamp);
    CREATE INDEX idx ON tbl(i, updated_at); -- it is not required to reproduce but make it to happen faster

Then it runs next scripts with pgbench concurrently:

    1) INSERT INTO tbl VALUES(13,now()) on conflict(i) do update set updated_at = now();
    2) INSERT INTO tbl VALUES(42,now()) on conflict(i) do update set updated_at = now();
    3) INSERT INTO tbl VALUES(69,now()) on conflict(i) do update set updated_at = now();

Also, during pgbench the next command is run in the loop:

    REINDEX INDEX CONCURRENTLY tbl_pkey;

For some time, everything looks more-less fine (except live locks, but this is the issue for the next test).
But after some time, about a minute or so (on ~3000th REINDEX) it just fails like this:

                make -C src/test/modules/test_misc/ check PROVE_TESTS='t/006_*'

                # waiting for an about 3000, now is 2174, seconds passed : 84
                # waiting for an about 3000, now is 2175, seconds passed : 84
                # waiting for an about 3000, now is 2176, seconds passed : 84
                # waiting for an about 3000, now is 2177, seconds passed : 84
                # waiting for an about 3000, now is 2178, seconds passed : 84
                # waiting for an about 3000, now is 2179, seconds passed : 84
                # waiting for an about 3000, now is 2180, seconds passed : 84
                # waiting for an about 3000, now is 2181, seconds passed : 84
                # waiting for an about 3000, now is 2182, seconds passed : 84
                # waiting for an about 3000, now is 2183, seconds passed : 84
                # waiting for an about 3000, now is 2184, seconds passed : 84

                #   Failed test 'concurrent INSERTs, UPDATES and RC status (got 2 vs expected 0)'
                #   at t/006_concurrently_unique_fail.pl line 69.

                #   Failed test 'concurrent INSERTs, UPDATES and RC stderr /(?^:^$)/'
                #   at t/006_concurrently_unique_fail.pl line 69.
                #                   'pgbench: error: pgbench: error: client 4 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(13) already exists.
                # client 15 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 9 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 11 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 8 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 3 script 2 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(69) already exists.
                # pgbench: error: client 2 script 2 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(69) already exists.
                # pgbench: error: client 12 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccold"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 10 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccold"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 18 script 2 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(69) already exists.
                # pgbench: error: pgbench:client 14 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey"
                # DETAIL:  Key (i)=(13) already exists.
                #  error: client 1 script 0 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey"
                # DETAIL:  Key (i)=(13) already exists.
                # pgbench: error: client 0 script 2 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey"
                # DETAIL:  Key (i)=(69) already exists.
                # pgbench: error: client 13 script 1 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(42) already exists.
                # pgbench: error: client 16 script 1 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(42) already exists.
                # pgbench: error: client 5 script 1 aborted in command 1 query 0: ERROR:  duplicate key value violates unique constraint "tbl_pkey_ccnew"
                # DETAIL:  Key (i)=(42) already exists.
                # pgbench: error: Run was aborted; the above results are incomplete.
                # '

Probably something wrong with arbiter index selection for different backends. I am afraid it could be a symptom of a more serious issue.

-------------------------------------

The second test shows an interesting live lock state in the similar situation.

    CREATE UNLOGGED TABLE tbl(i int primary key, n int);
    CREATE INDEX idx ON tbl(i, n);
    INSERT INTO tbl VALUES(13,1);

pgbench concurrently runs single command

    INSERT INTO tbl VALUES(13,1) on conflict(i) do update set n = tbl.n + EXCLUDED.n;

And also reindexing in the loop

    REINDEX INDEX CONCURRENTLY tbl_pkey;

After the start, a little bit strange issue happens

     make -C src/test/modules/test_misc/ check PROVE_TESTS='t/007_*'