I have three node Galera cluster setup with MariaDB, I see a repeating message for one of the node where it desyncs itself from cluster and resyncs within few seconds. This message is repeating every hour. Though it is just a notice, I would like to understand the reason. Configuration on all the servers is same. Does anyone has idea why this happens ? or if anybody faced similar problem of this sort. Below are the details.
MariaDB version 10.3.12
Galera version: 25.3.23-1
OS: RHEL 7.4
State transfer mechanism: default
Below is my Galera configuration on the node which has above mentioned observation.
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=300M; gcache.page_size=300M; pc.ignore_sb=false; pc.ignore_quorum=false"
#wsrep_cluster_address defines members of the cluster
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3
wsrep_cluster_name="mariadb-cluster"
#wsrep_node_address passed to replication pluggin
wsrep_node_address=192.168.1.1
#wsrep_node_incoming_address is used for replicationclient connections
wsrep_node_incoming_address=192.168.1.1
wsrep_debug=OFF
#
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
Error Message is as given below
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 0 [Note] WSREP: Member 2.0 (maria1) desyncs itself from group
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 7430165)
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 452 [Note] WSREP: Provider paused at 08e2494b-12ff-11e9-9785-4fa29d91e9c5:7430165 (22186)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 452 [Note] WSREP: resuming provider at 22186
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 452 [Note] WSREP: Provider resumed.
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Member 2.0 (maria1) resyncs itself to group
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 7430165)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Member 2.0 (maria1) synced with group.
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 7430165)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 2 [Note] WSREP: Synchronized with group, ready for connections
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
I could find out, I missed the fact that periodic backup was running on the node via Mariabackup tool. During that time node de-syncs itself from cluster for consistency.