On Tue, Jun 8, 2021 at 12:27 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > They're running this:
> > | PGOPTIONS="--deadlock_timeout=333ms -cstatement-timeout=3600s" psql -c "REINDEX INDEX CONCURRENTLY $i"
> > And if it times out, it then runs: $PSQL "DROP INDEX CONCURRENTLY $bad"
> ...
> > $ date -d @1623121264
> > Mon Jun 7 22:01:04 CDT 2021
Perhaps reindex was waiting on the VACUUM process to finish, while
VACUUM was (in effect) busy waiting on the REINDEX to finish. If the
bug is hard to reproduce then it might just be that the circumstances
that lead to livelock require that things line up exactly and the heap
page + XID level -- which I'd expect to be tricky to reproduce. As I
said upthread, I'm almost certain that the "goto retry" added by
commit 8523492d is a factor here -- that is what I mean by busy
waiting inside VACUUM. It's possible that busy waiting like this
happens much more often than an actual undetected deadlock/livelock.
We only expect to "goto retry" in the event of a concurrently aborting
transaction.
The other bug that you reported back in July of last year [1] (which
involved a "REINDEX INDEX pg_class_tblspc_relfilenode_index") was
pretty easy to recreate, just by running the REINDEX in a tight loop.
Could you describe how tricky it is to repro this issue now?
If you instrument the "goto retry" code added to lazy_scan_prune() by
commit 8523492d, then you might notice that it is hit in contexts that
it was never intended to work with. If you can reduce reproducing the
problem to reproducing hitting that goto in the absence of an aborted
transaction, then it might be a lot easier to produce a simple repro.
The livelock/deadlock is probably nothing more than the worst
consequence of the same issue, and so may not need to be reproduced
directly to fix the issue.
[1] https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com
--
Peter Geoghegan