Search code examples
elasticsearchsharding

Elasticsearch shard re-allocation

As far as I understand, shards in elasticsearch are distributed by an internal mechanism among all data nodes.

However,I have the following observations -

  • if we have not done any indexing and a new data node is added,the shards get distributed among the 2 nodes.
  • However,if some indexing has been done and thereafter,a new node is added then the shards are not distributed automatically. I have to move them manually via cluster rerouting.

Is it the expected behavior?

The issue at hand is that I started out with just one node,5 shards and no replicas. The index contains about 100 GB of data . Now i want to add a new data node.

My questions -

  • Does elasticsearch take some time before deciding to move shards across nodes?
  • For clusters with index data already present,is manual cluster rerouting the only way to move shards?

Note - Cluster rerouting works fine .However, these questions will help me plan for future.

TIA

Solution

  • If you need to distribute shards right after you add a node, then you have to execute rerouting manually. Even though you don't run it manually, after time elasticsearch distributes your shards evenly.

    There some configuration options you can use regarding shard rebalancing

    in index settings

    "index.routing.allocation.total_shards_per_node": 1

    in `cluster settings

    "transient":{
    "cluster.routing.allocation.cluster_concurrent_rebalance": 1
}

    Configuration parameters themselves explain what they exist for, you can modify values and see how they work for you.