How to prevent shard relocation from hot nodes to warm or cold nodes?

Question

What do I have:

• Elasticsearch (7.7.0) cluster (amazon/opendistro-for-elasticsearch:1.8.0 Docker image)
  • one master node
  • one coordinating node
  • two data nodes with node.attr.data=hot
  • one data node with node.attr.data=warm

What do I want: prevent shard allocation and relocation from hot data nodes to warm (and cold in future) data nodes.

What have I tried:

I've put "index.routing.allocation.require.data": "hot" for all index templates, so newly created indices won't be allocated to any but hot data nodes. This works fine.

Anyway, I can't restrict shards relocation from hot nodes to warm. At the moment I'm using "cluster.routing.allocation.exclude._ip" : "warm data node ip" to prevent relocation from hot data nodes to the warm data node. But will I be able to use ILM with this filter?

I've also tried to

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.awareness.attributes": ["data"]
  }
}

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.awareness.force.data.values": ["hot"]
  }
}

and then remove the "cluster.routing.allocation.exclude._ip" filter. Shards were relocating from hot data nodes to the warm data node anyway. What am I missing?

Answer 1

I had to update settings for my old incides:

PUT my-index/_settings
{
  "index.routing.allocation.require.data": "hot"
}

Indices which are under certain index template won't be updated if the index template is updated.