Search code examples
docker-composeclickhouse

Simple clickhouse cluster via clickhouse-keeper in Docker

I'm trying to set up and test replication via clickhouse-keeper.

This is my docker.compose.yaml

version: "3.7"

services:
  clickhouse01:
    image: yandex/clickhouse-server
    container_name: clickhouse01
    hostname: clickhouse01
    ports:
      - 8811:8123
      - 9011:9000
    volumes:
      - "D:/clichhouse-data/volume1:/var/lib/clickhouse"

  clickhouse02:
    image: yandex/clickhouse-server
    hostname: clickhouse02
    container_name: clickhouse02
    ports:
      - 8812:8123
      - 9012:9000
    volumes:
      - "D:/clichhouse-data/volume2:/var/lib/clickhouse"

This is remote_servers section that I'm using on each node in /etc/clickhouse-server/config.xml

<remote_servers>
        <test_cluster_two_shards_localhost>
             <shard>
                 <replica>
                     <host>clickhouse01</host>
                     <port>9000</port>
                 </replica>
                 <replica>
                    <host>clickhouse02</host>
                    <port>9000</port>
                </replica>
             </shard>
        </test_cluster_two_shards_localhost>
    </remote_servers>

This is keeper.xml file that I'm using on the 1st node, path - /etc/clickhouse-server/config.d/:

<clickhouse_keeper>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>

        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse02</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse_keeper>

This is keeper.xml file that I'm using on the 2nd node, path - /etc/clickhouse-server/config.d/:

<clickhouse_keeper>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>2</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>

        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse02</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse_keeper>

This is macros.xml file that I'm using on the 1st node, path - /etc/clickhouse-server/config.d/:

<clickhouse>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <macros>
        <cluster>test_cluster_two_shards_localhost</cluster>
        <replica>replica1</replica>
        <shard>1</shard>
    </macros>
</clickhouse>

This is macros.xml file that I'm using on the 2nd node, path - /etc/clickhouse-server/config.d/:

<clickhouse>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <macros>
        <cluster>test_cluster_two_shards_localhost</cluster>
        <replica>replica2</replica>
        <shard>1</shard>
    </macros>
</clickhouse>

This is use-keeper.xml file that I'm using on each node, path - /etc/clickhouse-server/config.d/:

<clickhouse>
    <zookeeper>
        <node index="1">
            <host>clickhouse01</host>
            <port>9181</port>
        </node>
        <node index="2">
            <host>clickhouse02</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>

I can see expected cluster in system.clusters table system.clusters query

But I don't have system.zookeeper table system.zookeeper query

I also tried to check if keeper is running by echo ruok | nc 127.0.0.1 9181 - no answer.

References:

  1. https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-zookeeper/clickhouse-keeper/
  2. https://mrkaran.dev/posts/clickhouse-replication/
Solution

  • Resolved.

    My mistake was that I copied the entire configuration after running the containers by using cp command in the following way: docker cp 'C:\Users\{user}\Desktop\CH cluster\node-1\config.xml' clickhouse01:/etc/clickhouse-server

    I thought that I need to run clickhouse-server with default configuration and then restart the clickhouse-server inside container after applying new configuration.

    So that, it was the issue with my understanding how I should apply and change configuration.

    Changes:

    1. I placed locally (D:/clichhouse-data/node-1) all need configs (keeper.xml, macros.xml and etc.)
    2. Updated docker-compose.yaml file in the following way:
    version: "3.7"

services:
  clickhouse01:
    image: yandex/clickhouse-server
    container_name: clickhouse01
    hostname: clickhouse01
    ports:
      - 8811:8123
      - 9011:9000
      - 9181:9181
    volumes:
      - type: volume
        source: clickhouse01
        target: /var/lib/clickhouse
      - "D:/clichhouse-data/node-1/config.xml:/etc/clickhouse-server/config.xml"
      - "D:/clichhouse-data/node-1:/etc/clickhouse-server/config.d/"

  clickhouse02:
    image: yandex/clickhouse-server
    hostname: clickhouse02
    container_name: clickhouse02
    ports:
      - 8812:8123
      - 9012:9000
      - 9182:9181
    volumes:
      - type: volume
        source: clickhouse02
        target: /var/lib/clickhouse
      - "D:/clichhouse-data/node-2/config.xml:/etc/clickhouse-server/config.xml"
      - "D:/clichhouse-data/node-2:/etc/clickhouse-server/config.d/"
      
volumes:
  clickhouse01:
  clickhouse02:

    And as a result configuration updated on the startup.