On Wed, Mar 7, 2018 at 12:01 PM, Peter Geoghegan <pg@bowt.ie> wrote:
I happen to know that bricklen already ran amcheck. There were errors, but they were not consistent with a collation issue. Rather, it looked like something was up with the storage layer -- the sibling links of a pair of pages were not in mutual agreement.
Even if that wasn't something that I knew already, I still would not suspect opclass misbehavior of any variety. VACUUM doesn't care about the ordering of items on the page in the case of nbtree. And, it performs a physical order scan there (albeit with some extra trickery to prevent races due to concurrent splits). Index tuples that could end up being unreachable to index scans due to opclass misbehavior should remain reachable to VACUUM.
What little detail I've been able to collect so far is below. All for 10.1 clusters.
From the postgres logs, for 6 different databases (across 3 geo regions, of which two were on the same hypervisor). Each one was discovered when autovacuum tried to vacuum them:
ERROR: could not find left sibling of block 4775 in index "<some index>" ERROR: right sibling 13983 of block 7196 is not next child 7246 of block 5208 in index "<some index>" ERROR: right sibling 60252 of block 60115 is not next child 60118 of block 60113 in index "<some index>" ERROR: right sibling 93058 of block 93057 is not next child 93061 of block 93008 in index "<some index>" ERROR: right sibling 10081 of block 10079 is not next child 10084 of block 10046 in index "<some index>" ERROR: left link changed unexpectedly in block 13868 of index "<some index>" ERROR: right sibling 145 of block 92 is not next child 93 of block 3 in index "<some index>"
A strace from the hung autovac process (before we killed it):