I'm trying to set up ElasticSearch (woth the ECK operator) so that it has nodes running on cheap machines (for "normal" flow), and nodes running on "expensive" machines (for burst moments). It is important to note that the data I use is completely ingested and there is no indexing done with the queries (only search queries happen). I only have one index, so I'll skip that section in the diagrams.
I decided to choose the following approach: Have the data ingested on a "cheap" node (CN), which will serve as the primary node for my shards. Master nodes run there too, but this isn't relevant (I think).
+--------------------- Cheap Node -------------------------+
| [shard0_P] [shard1_P] [shard2_P] [shard3_P] [shard4_P] |
+----------------------------------------------------------+
I now also want "expensive" nodes running on bigger machines. I want 2 nodes (EN1, EN2), and they can share the shards. I want to achieve this:
+-------------------- Expensive Node1 ---------------------+
| [shard0_R] [shard1_-] [shard2_R] [shard3_-] [shard4_R] |
+----------------------------------------------------------+
+-------------------- Expensive Node2 ---------------------+
| [shard0_-] [shard1_R] [shard2_-] [shard3_R] [shard4_-] |
+----------------------------------------------------------+
Shards will be evenly distributed on EN1 and EN2.
The reason for having the primary shards all located on CN is that I want to be able to turn off all the expensive nodes and still have all my data.
I tried running the clusted with only CN up, and ingesting the data there. I then set cluster.routing.rebalance.enable=replicas to avoid having the shards moving out of CN into ENs, and switched on the nodes.
My data_1 and _data2 pods were allocated on the EN nodes, but there was no mirroring of the shards. When I tried changing the value of the cluster.routing.rebalance.enable to something else (primaries), the shards started leaving the CN node to even out everything on the EN nodes - not what I wanted :(
Is there a way to configure ElasticSearch to achieve this design ?
The reason for having the primary shards all located on CN is that I want to be able to turn off all the expensive nodes and still have all my data.
If you turn off your expensive nodes, you still have all your data.
If you turn off your cheap node, you still have all your data.
Since you're only doing searches, it makes no difference whether your shards are primaries or replicas on the cheap node. If you lose a primary shard because you turn off a node that hosts it, then a replica will be promoted to primary. Primary and replicas shards are exact copies of each other, sort of like gemini.