I am getting below error in my 2 node BDR setup. postgres going down. any idea?
<35382016-06-07 10:16:59 GMT%LOG: database system was interrupted; last known up at 2016-06-07 09:06:44 GMT
<35382016-06-07 10:16:59 GMT%PANIC: replication slot file "pg_replslot/bdr_16389_6293051490331141125_2_16389__/state" has
wrong magic 4522536 instead of 17112225
<35352016-06-07 10:16:59 GMT%LOG: startup process (PID 3538) was terminated by signal 6: Abort trap
<35352016-06-07 10:16:59 GMT%LOG: aborting startup due to startup process failure
That suggests that there was a write failure on the replication slot file.
A simple write error shouldn't be possible because we write the slot file to a tempfile, then replace the old slot file with the new one. Filesystem issues are possible, or memory corruption in the application that caused a bad write. Or a bug, but it's hard to see how we could write the wrong slot magic number here.
With the slot corrupted all you can really do is part one of the nodes then join a new one.
If you're able to reproduce this I'd really like to see how it came about.