Re: BDR Rejoin of failed node, hangs.

Good Morning,

Has anyone had a moment to look at this?
It is a bit of a show stopper.

Thanks Steve

________________________________________
From: pgsql-general-owner@postgresql.org <pgsql-general-owner@postgresql.org> on behalf of Steve Pribyl
<spribyl@akunacapital.com>
Sent: Thursday, October 1, 2015 12:27 PM
To: pgsql-general@postgresql.org
Subject: [GENERAL] BDR Rejoin of failed node, hangs.

Good Afternoon,

I am in the process of testing out BDR and am having problems rejoining after a simulated loss of one node.   The join
hangswhile waiting to complete and there interesting errors in the logs. 


Simulation process
After creating the database on both nodes.

Shutdown postgres and reset db on node 2
   rm -rf /var/lib/postgresql/9.4/main/*
   /usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main -A trust -U postgre

Clean up node 1.
   select bdr.bdr_part_by_node_names('{node2}');
   delete from bdr.bdr_nodes where node_status='k';

When I try to re-add node2
Using the NODE 2 Create sql again, it hangs
SELECT bdr.bdr_node_join_wait_for_ready(); hangs.

The log on node2  has errors
http://pastebin.com/8ZsTe5cG
55000: System identification mismatch between connection and slot
00000: worker process: bdr db: bdrdemo (PID 12042) exited with exit code 1

The log on node1 does not have any errors.
http://pastebin.com/njVJ9WX7

Both nodes show up in select * from bdr.bdr_nodes; on node1 and node2;

Database Creation:
NODE 1
create database bdrdemo;
\connect bdrdemo
CREATE EXTENSION btree_gist;
CREATE EXTENSION bdr;
select bdr.bdr_group_create(local_node_name := 'node1', node_external_dsn := 'host=192.168.101.41 port=5432
dbname=bdrdemo');
SELECT bdr.bdr_node_join_wait_for_ready();

NODE 2
create database bdrdemo;
\connect bdrdemo
CREATE EXTENSION btree_gist;
CREATE EXTENSION bdr;
select bdr.bdr_group_join(local_node_name := 'node2', node_external_dsn := 'host=192.168.101.42 port=5432
dbname=bdrdemo',join_using_dsn := 'host=192.168.101.41 port=5432 dbname=bdrdemo'); 
SELECT bdr.bdr_node_join_wait_for_ready();


Ubuntu 14.04 packages
ii  postgresql-bdr-9.4                  9.4.4-1trusty                 amd64        object-relational SQL database,
version9.4 server 
ii  postgresql-bdr-9.4-bdr-plugin       0.9.2-1trusty                 amd64        BDR Plugin for PostgreSQL-BDR 9.4
ii  postgresql-bdr-client-9.4           9.4.4-1trusty                 amd64        front-end programs for
PostgreSQL-BDR9.4 
ii  postgresql-bdr-contrib-9.4          9.4.4-1trusty                 amd64        additional facilities for PostgreSQL
ii  postgresql-client-common            154                           all          manager for multiple PostgreSQL
clientversions 
ii  postgresql-common                   154                           all          PostgreSQL database-cluster manager

TIA,
Steve
________________________________
 [http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com>
p: +1 312 994 4646 | m:  | f: +1 312 750 1667 | spribyl@akunacapital.com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or
otherwiseprotected from disclosure. This information is intended for the use of the addressee only and is not offered
asinvestment advice to be relied upon for personal or professional use. Additionally, all electronic messages are
recordedand stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are
herebynotified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on,
thecontents of this electronic message is strictly prohibited. If you have received this communication in error, please
notifyus by telephone at (312)994-4640 and destroy the original message. 


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
________________________________
 [http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com>
p: +1 312 994 4646 | m:  | f: +1 312 750 1667 | spribyl@akunacapital.com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or
otherwiseprotected from disclosure. This information is intended for the use of the addressee only and is not offered
asinvestment advice to be relied upon for personal or professional use. Additionally, all electronic messages are
recordedand stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are
herebynotified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on,
thecontents of this electronic message is strictly prohibited. If you have received this communication in error, please
notifyus by telephone at (312)994-4640 and destroy the original message.