> I restored a snapshot and I can drop the tables there, so we'll likely > proceed to swap the replicas over tomorrow. I have this corrupted > instance i can continue to debug on if necessary. There seem to be some > other issues now that we're investigating, like a max(timestamp) query on > the old instance is doing a seqscan on all the partitions instead of using > an index that's there, the new instance from the snapshot correctly uses > the index, so I suspect further corruption.
This being RDS makes it very difficult to debug. For example I would suggest a debugger to find the problem, but you don't have access to that.
Yes agreed, we'll probably terminate the instance today if there's no further lines of investigation. I know this isn't their forum, but I don't get why AWS pins all their extension versions to the postgres version, If you use 12.2 on rds, you get pglogical 2.3.0. If you want 2.3.1, you have to upgrade to pg 12.3. I don't get that logic, shouldn't 2.3.1 be the current version on both? What's the point of them supporting 12.2 if they ship an extension version that 2ndQ says you shouldn't use?
Anyway, just musing out loud, don't expect any kind of answer to that here. :)