Oops, i missed including pgsql-general in my earlier replies..
> > > I have data_checksums switched on so am suspecting a streaming > > > > replication bug. Anyone know of a recent bug which could have caused > > > > this? > > > > > > > > > > I cannot conclude at this point. I encountered these kind of errors with > > > Indexes and re-indexing fixed them. > > > > This is actually the second time I am seeing these kinds of errors, in > > the past, after verifying that no data was lost I used VACUUM FULL to > > recover the ability to INSERT. There was no pitchfork uprising... > > > > Did you check your disk file systems for any issues ?
I'm on EC2 and praying that data_checksums takes care of that for me...
Please check if any issues with the storage.
> These errors only > happen after you do pg_restore ?
they happened after the failover, I'm not convinced they were caused by the pg_restore.
mmh ok. Any issues on slave before failover ?
I still have the backups and WAL logs and could probably try use them to build a bad slave. But that's a few hundred GB of WAL and is not worth it unless I know what to look for.
Well, that is the work around if you are encountering data loss due to this bug.
Developers must be able to tell you if this is a bug in streaming replication. I haven't encountered such bug till now.