I have deployed MariaDB Cluster before, and this problem only comes out recently (I don't have this problem before and I don't know why).
I have server 1, 2 and 3. I executed an INSERT command at server 3, however, the tables at server 1 and 2 remains unchanged.
3 servers are at different parts of the world. After the INSERT command, the state uuid remains the same.
Here is the status of server 1:
MariaDB [mysql]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 205 |
| wsrep_replicated | 170 |
| wsrep_replicated_bytes | 160481 |
| wsrep_repl_keys | 664 |
| wsrep_repl_keys_bytes | 9222 |
| wsrep_repl_data_bytes | 140379 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 46 |
| wsrep_received_bytes | 26150 |
| wsrep_local_commits | 170 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 1 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_cached_downto | 1 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.482927 |
| wsrep_apply_oooe | 0.009756 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.009756 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.009756 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0.200155/0.201113/0.201752/0.000614937/4 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | c4f91b4f-fee1-11e5-8c4f-6e451c332f79 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 6 |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <[email protected]> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
Status of server 2:
MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 225 |
| wsrep_replicated | 35 |
| wsrep_replicated_bytes | 25700 |
| wsrep_repl_keys | 119 |
| wsrep_repl_keys_bytes | 1757 |
| wsrep_repl_data_bytes | 21703 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 187 |
| wsrep_received_bytes | 177793 |
| wsrep_local_commits | 35 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 1 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 4 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.032086 |
| wsrep_local_cached_downto | 9 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.193548 |
| wsrep_apply_oooe | 0.004630 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.004630 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.009217 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0.200138/0.201917/0.203696/0.00177914/2 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | d562e272-fee1-11e5-b2a2-d3a6b5579aab |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 1 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <[email protected]> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
57 rows in set (0.01 sec)
Status of server3 (As you can see, the latency shows all 0 but I don't know why)
MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 245 |
| wsrep_replicated | 5 |
| wsrep_replicated_bytes | 4350 |
| wsrep_repl_keys | 11 |
| wsrep_repl_keys_bytes | 203 |
| wsrep_repl_data_bytes | 3827 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 226 |
| wsrep_received_bytes | 208559 |
| wsrep_local_commits | 1 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_cached_downto | 19 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.022026 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.008811 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | fd022144-fee1-11e5-a7a3-f23274fef9c3 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 2 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <[email protected]> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
57 rows in set (0.00 sec)
iptables at all three servers are set to ACCEPT all input and output traffics.
The log shows that all servers have joined and synced with the cluster.
Does anyone know why? Thanks.
I finally found that is is the app's problem that use MyISAM as storage engine, which causes the error. There is no error after change back to InnoDB.