Search code examples
dockerdocker-composemariadbgalera

MariaDB galera cluster 10.2 second node "Failed to open channel"


I'm trying to run mariadb:10.2.14 as a galera cluster on my windows computer locally using docker compose. Running the initial boot node works fine, but the second node fails to join the cluster with an error:

node_1 | 2018-05-04 3:13:46 140187778701184 [Note] WSREP: view((empty))

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():158

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'galera' at 'gcomm://boot': -110 (Connection timed out)

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] WSREP: gcs connect failed: Connection timed out

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] WSREP: wsrep::connect(gcomm://boot) failed: 7

node_1 | 2018-05-04 3:13:46 140187778701184 [ERROR] Aborting

I ran ping boot in the container to verify that the host name resolved correctly, not sure why it is unable to connect. I tried to base the config off of various docker files I see for mariadb:10.1 such as https://gist.github.com/lucidfrontier45/497341c4b848dfbd6dfb

My docker compose file:

# Docker compose file for running a local MySQL server
version: '2.2'
services:
  boot:
    image: mariadb:10.2.14
    command: mysqld --user=mysql --wsrep_new_cluster
    environment:
      MYSQL_DATABASE: "db"
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
      # Needed because galera doesn't support MyISAM, which tzinfo uses
      MYSQL_INITDB_SKIP_TZINFO: "yes"
    ports:
      - ${SQL_PORT}:3306
      - 4444:4444
      - 4567:4567
      - 4568:4568
    networks:
      - sql
    volumes:
      - ./kubernetes/mariadb.conf.d:/etc/mysql/mariadb.conf.d
      - /var/lib/mysql
  node:
    image: mariadb:10.2.14
    command: mysqld --user=mysql --wsrep_cluster_address=gcomm://boot
    environment:
      MYSQL_DATABASE: "db"
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
      # Needed because galera doesn't support MyISAM, which tzinfo uses
      MYSQL_INITDB_SKIP_TZINFO: "yes"
    networks:
      - sql
    volumes:
      - ./kubernetes/mariadb.conf.d:/etc/mysql/mariadb.conf.d
      - /var/lib/mysql
networks:
  sql:

My config file in maraidb.conf.d:

# This will be passed to all mysql clients
[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4

# The MySQL server
[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
default_storage_engine=innodb
binlog_format=row
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0

# Allow server to accept connections on all interfaces.
bind-address=0.0.0.0

#
# * Galera-related settings
#
# https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/
#
[galera]
wsrep_on=ON
wsrep_log_conflicts=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# TODO: is rsync the best option?
wsrep_sst_method=rsync

wsrep_cluster_name=galera
#wsrep_slave_threads=1

Solution

  • Thanks @Nickolay, your answer mostly works, but I ran into another problem. It did set me on the right track to find the solution though.

    So it seems the main problem was that --wsrep_new_cluster by itself isn't enough to bootstrap the node, you need to set the wsrep_cluster_address variable. Setting it using --wsrep_cluster_address=gcomm:// did the trick for me.

    Also I ran into problems where there seemed to be a race condition and the first boot node would fail to initialize with an error that it wasn't the last node. I fixed this by using a short sleep in the second node command.

    My final docker compose file:

    # Docker compose file for running a local mariadb galera cluster
    version: '3.6'
    services:
      boot:
        image: mariadb:10.2.14
        command: mysqld --user=mysql --wsrep_cluster_address=gcomm://
        environment:
          MYSQL_DATABASE: "db"
          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
          # Needed because galera doesn't support MyISAM, which tzinfo uses
          MYSQL_INITDB_SKIP_TZINFO: "yes"
        ports:
          - ${SQL_PORT}:3306
          - 4444:4444
          - 4567:4567
          - 4568:4568
        volumes:
          - ./kubernetes/mariadb.conf.d:/etc/mysql/mariadb.conf.d
          - /var/lib/mysql
      node:
        image: mariadb:10.2.14
        command: bash -c "sleep 10; mysqld --user=mysql --wsrep_cluster_address=gcomm://boot"
        environment:
          MYSQL_DATABASE: "db"
          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
          # Needed because galera doesn't support MyISAM, which tzinfo uses
          MYSQL_INITDB_SKIP_TZINFO: "yes"
        volumes:
          - ./kubernetes/mariadb.conf.d:/etc/mysql/mariadb.conf.d
          - /var/lib/mysql