I have a Galera cluster running on MariaDB containers, and only the bootstrapping node accepts connections - the other ones don't create a socket.
I'm baffled by this behavior since all the nodes are created from the same VM template and have the same configuration (apart from wsrep_cluster_address = gcomm:// in the bootstrapper node).
I'm using mariadb/server:10.4 as the container image.
/etc/mysql/my.cnf (the only configuration file that exists and is the same across all nodes):
[mysqld]
port = 3306
report_host = {{ ansible_host }} # always FQDN
default_storage_engine = InnoDB
binlog_format = ROW
character-set-server = utf8mb4
character_set_server = utf8mb4
[mariadb]
bind-address = 0.0.0.0
innodb_autoinc_lock_mode = 2
# Galera cluster configuration
wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_address = gcomm://10.0.0.100,10.0.1.101,10.0.0.102,10.0.0.103
wsrep_cluster_name = my-galera-cluster
wsrep_sst_method = rsync
wsrep_sst_auth = root:s3cr3t-r00t-p4ssw0rd
# Cluster node configuration
wsrep_node_address = {{ inventory_hostname }} # always IP
wsrep_node_name = {{ ansible_host.split(".")[0] | lower }} # always host name
This is how I run each node in Ansible:
- name: Start MariaDB docker container
docker_container:
image: "mariadb/server:10.4"
name: "mariadb-server"
state: started
log_driver: json-file
log_options:
max-size: "100m"
max-file: "2"
env:
MARIADB_ROOT_PASSWORD_FILE: "/etc/mysql/secrets/mariadb_root_password"
MARIADB_USER: "my-user"
MARIADB_PASSWORD_FILE: "/etc/mysql/secrets/mariadb_my-user_password"
MARIADB_ALLOW_EMPTY_PASSWORD: "no"
MARIADB_ROOT_HOST: "%"
# https://github.com/docker-library/mariadb/issues/262
MYSQL_INITDB_SKIP_TZINFO: "1"
volumes:
- "/etc/mysql:/etc/mysql"
- "/var/lib/mysql:/var/lib/mysql"
ports:
- "{{ inventory_hostname }}:3306:3306"
- "{{ inventory_hostname }}:4567:4567"
- "{{ inventory_hostname }}:4567:4567/udp"
- "{{ inventory_hostname }}:4568:4568"
- "{{ inventory_hostname }}:4444:4444"
When I check the cluster status on the bootstrapping node I get happy figures such as:
wsrep_cluster_size 4
wsrep_cluster_status Primary
wsrep_connected ON
wsrep_ready ON
The mysqld daemon is present on all nodes:
# docker exec mariadb-server ps -ef
UID PID PPID C STIME TTY TIME CMD
mysql 1 0 0 07:12 ? 00:00:06 mysqld
mysql 46 1 0 07:12 ? 00:00:00 sh -c wsrep_sst_rsync --role 'joiner' --address '10.0.1.102' --datadir '/var/lib/mysql/' --parent '1'
mysql 47 46 0 07:12 ? 00:00:04 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address 10.0.1.102 --datadir /var/lib/mysql/ --parent 1
mysql 98 47 0 07:12 ? 00:00:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
mysql 16029 47 0 08:18 ? 00:00:00 sleep 1
root 16030 0 0 08:18 ? 00:00:00 ps -ef
Each of the containers has the default socket directory: /var/run/mysqld, yet only the bootstrapping node creates mysqld.sock, so trying to log in locally fails:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Even when I add socket = /var/run/mysqld/mysqld.sock to any section ([mariadb], [mysql], [mysqld], [client]) it seems that the other nodes just skip its creation. The bootstrapping node is the only that produces this line in the log:
[Note] Server socket created on IP: '0.0.0.0'
How to make the other nodes create the socket too? Important note: when I disable the Galera configuration each node creates a socket successfully.
I switched to directly installing MariaDB on the operating system with no containers and my issue with the socket was resolved... Strange.