I have a cluster of Postgres BDR that has 3 nodes "Ready" and 3 nodes "Parted".
If I do SELECT * FROM bdr.bdr_nodes the following information is displayed:
-[ RECORD 1 ]------+-------------------------
node_sysid | 6153716379158074503
node_timeline | 1
node_dboid | 16385
node_status | r
node_name | node3
node_local_dsn | host=x.x.x.241 [...]
node_init_from_dsn | host=x.x.x.47 [...]
-[ RECORD 2 ]------+-------------------------
node_sysid | 6153716914784688297
node_timeline | 1
node_dboid | 16385
node_status | r
node_name | node2
node_local_dsn | host=x.x.x.5 [...]
node_init_from_dsn | host=x.x.x.47 [...]
-[ RECORD 3 ]------+-------------------------
node_sysid | 6170758438846557459
node_timeline | 1
node_dboid | 16384
node_status | r
node_name | node4
node_local_dsn | host=x.x.x.128 [...]
node_init_from_dsn | host=x.x.x.47 [...]
-[ RECORD 4 ]------+-------------------------
node_sysid | 6153716402564903569
node_timeline | 1
node_dboid | 16385
node_status | k
node_name | node1
node_local_dsn | host=x.x.x.47 [...]
node_init_from_dsn |
-[ RECORD 5 ]------+-------------------------
node_sysid | 6170830020100809103
node_timeline | 1
node_dboid | 16385
node_status | k
node_name | node6
node_local_dsn | host=x.x.x.48 [...]
node_init_from_dsn | host=x.x.x.241 [...]
-[ RECORD 6 ]------+-------------------------
node_sysid | 6170839982079996801
node_timeline | 1
node_dboid | 16385
node_status | c
node_name | node8
node_local_dsn | host=x.x.x.142 [...]
node_init_from_dsn | host=x.x.x.241 [...]
-[ RECORD 7 ]------+-------------------------
node_sysid | 6170833985333433816
node_timeline | 1
node_dboid | 16385
node_status | k
node_name | node7
node_local_dsn | host=x.x.x.48 [...]
node_init_from_dsn | host=x.x.x.241 [...]
I am trying to join node8. But it just won't happen. The error is the following:
d= p=5521 a=ERROR: 08006: could not connect to the primary server: could not connect to server: Connection timed out
Is the server running on host "x.x.x.48" and accepting
TCP/IP connections on port 5432?
d= p=5521 a=DETAIL: Connection string is 'host=x.x.x.48 [...]'
That error means that it is trying to connect to a node that has been killed or removed. Why is trying to connect to a node that is killed or removed ? How can I fix this situation ?
The following command was used to join node8
SELECT bdr.bdr_group_join(
local_node_name := 'node8',
node_external_dsn := 'host=x.x.x.142 [...]',
join_using_dsn := 'host=x.x.x.241 [...]'
);
BDR has been installed according to this instructions (Debian Wheezy):
curl -sSL https://manageacloud.com/api/cm/configuration/postgresql-bdr/debian/manageacloud-production-script.sh | bash
Table bdr.bdr_connections:
-[ RECORD 1 ]----------+---------------------
conn_sysid | 6170839982079996801
conn_timeline | 1
conn_dboid | 16385
conn_origin_sysid | 0
conn_origin_timeline | 0
conn_origin_dboid | 0
conn_is_unidirectional | f
conn_dsn | host=x.x.x.142 [...]
conn_apply_delay |
conn_replication_sets | {default}
-[ RECORD 2 ]----------+----------------------
conn_sysid | 6153716402564903569
conn_timeline | 1
conn_dboid | 16385
conn_origin_sysid | 0
conn_origin_timeline | 0
conn_origin_dboid | 0
conn_is_unidirectional | f
conn_dsn | host=x.x.x.47 [...]
conn_apply_delay |
conn_replication_sets | {default}
-[ RECORD 3 ]----------+-----------------------
conn_sysid | 6153716379158074503
conn_timeline | 1
conn_dboid | 16385
conn_origin_sysid | 0
conn_origin_timeline | 0
conn_origin_dboid | 0
conn_is_unidirectional | f
conn_dsn | host=x.x.x.241 [...]
conn_apply_delay |
conn_replication_sets | {default}
-[ RECORD 4 ]----------+-----------------------
conn_sysid | 6153716914784688297
conn_timeline | 1
conn_dboid | 16385
conn_origin_sysid | 0
conn_origin_timeline | 0
conn_origin_dboid | 0
conn_is_unidirectional | f
conn_dsn | host=x.x.x.5 [...]
conn_apply_delay |
conn_replication_sets | {default}
-[ RECORD 5 ]----------+-----------------------
conn_sysid | 6170758438846557459
conn_timeline | 1
conn_dboid | 16384
conn_origin_sysid | 0
conn_origin_timeline | 0
conn_origin_dboid | 0
conn_is_unidirectional | f
conn_dsn | host=x.x.x.128 [...]
conn_apply_delay |
conn_replication_sets | {default}
Version:
# SELECT bdr.bdr_version();
bdr_version
-------------------
0.9.1-2015-05-26-
(1 row)
This is a bug in BDR. I've just fixed it in my local copy of the bdr-plugin/next tree and will push the change to bdr-plugin/REL0_9_STABLE for inclusion in 0.9.3 once I've tested it locally.
The issue is that we weren't filtering out bdr.bdr_connections rows based on the bdr.bdr_nodes.state during creation of slots on peers as part of node join.
It is safe to delete any bdr.bdr_connections entry with no corresponding bdr.bdr_nodes entry, or where the bdr.bdr_nodes entry has state = 'k' to work around this in 0.9.2 and older.