Search code examples
etlclickhouse

Load data into clickhouse from other clickhouse servers

I need to load data into ClickHouse server from other ClickHouse servers.

I have four ClickHouse servers which have 25000 inserts/server/second approx. (100000 inserts/sec). I want to load data from all these database servers to a fifth server.

I have looked into table engines like URL or JDBC bridge but I was wondering what options do I have, for continuous push / pull or scheduled loads. Using files is not possible because of the large size and extra work. DBs are busy 24x7. What are the possible solutions. Thanks in advance.

Solution

  • I would create tables with engine=Distributed on those 4 servers and point to it 5-th server, using remote_servers cluster description. And using Materialized View will intercept/duplicate inserts.