As a dependent question to an existing open thread, where I was doubting the strategy or implementation details of settings when modified for a cluster as well as an index. While I had tried to configure per-index level allocation, the configuration is not really working dynamically as expected.
Our current configurations are reflecting upto date after executing the relevant commands:
GET user/_settings?include_defaults=true&filter_path=**.routing.**
{
"user": {
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"size": "big",
"rack": "compute"
},
"exclude": {
"rack": "storage"
}
}
}
}
},
"defaults": {
"index": {
"routing": {
"rebalance": {
"enable": "ALL"
},
"allocation": {
"enable": "ALL",
"total_shards_per_node": "-1"
}
}
}
}
}
}
with cluster settings :
GET _cluster/settings?include_defaults=true&filter_path=**.routing.**
{
"transient": {
"cluster": {
"routing": {
"rebalance": {
"enable": "all"
},
"allocation": {
"disk": {
"watermark": {
"low": "90%"
}
},
"enable": "all",
"total_shards_per_node": "1000"
}
}
}
},
"defaults": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_incoming_recoveries": "2",
"disk": {
"threshold_enabled": "true",
"watermark": {
"high": "90%"
},
"include_relocations": "true",
"reroute_interval": "60s"
},
"node_initial_primaries_recoveries": "4",
"awareness": {
"attributes": ""
},
"balance": {
"index": "0.55",
"threshold": "1.0",
"shard": "0.45"
},
"same_shard": {
"host": "false"
},
"node_concurrent_outgoing_recoveries": "2",
"allow_rebalance": "indices_all_active",
"cluster_concurrent_rebalance": "2",
"node_concurrent_recoveries": "2",
"type": "balanced",
"snapshot": {
"relocation_enabled": "false"
}
}
}
}
}
}
The reason for me to affirm to this conclusion is that nodes mentioned in _cat/shards for the user index involves all types of nodes, even with labels excluded!
user 1 r STARTED ... ... search02
user 1 p STARTED ... ... search05
user 1 r STARTED ... ... search22
user 3 r STARTED ... ... search21
user 3 r STARTED ... ... search10
user 3 p STARTED ... ... search26
user 2 r STARTED ... ... search09
user 2 r STARTED ... ... search08
user 2 p STARTED ... ... search11
user 4 p STARTED ... ... search06
user 4 r STARTED ... ... search15
user 4 r STARTED ... ... search27
user 0 r STARTED ... ... search35
user 0 p STARTED ... ... search36
user 0 r STARTED ... ... search20
Q: Is there something that I am missing out? Or does this change need a rolling restart of my data nodes(pods in K8s) cluster for the changes to reflect upon?
Update: As asked in the comment by Val
GET _cat/nodeattrs?v
node host ip attr value
search17 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search20 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search13 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search15 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search28 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search01 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search21 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search12 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search31 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search19 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search08 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search35 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search10 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search14 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search27 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search36 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search33 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search09 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search16 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search29 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search24 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search11 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search18 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search07 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search34 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search23 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search22 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search26 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search37 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search06 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search32 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search04 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search02 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search03 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search00 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search05 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search30 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
search25 10.xx.xx.xx 10.xx.xx.xx ml.enabled true
If you look at the output of GET _cat/nodeattrs?v, you can see if your node attributes have been taken into account after modifying your configuration file.
Even though you can change index and cluster level settings related to shard allocation filtering dynamically, modifying node attributes requires restarting the nodes. The reason it is not possible to set node attributes dynamically is because the configuration file is not reloadable upon change, and there's no API that allows to change node settings/attributes dynamically, only cluster and index settings.
What I would do in your case if to use index-level shard filtering for the user index and specify index.routing.allocation.require instead of index.routing.allocation.include, the latter is just an hint, while the former is a hard constraint.
That way you can force the user index to be on big nodes and other index to not be on big nodes.