Search code examples
elasticsearchaws-elasticsearch

AWS Elasticsearch frequently getting into yellow state


  • I have AWS Elasticsearch running with 3 Master nodes(C4.large) and 10 data nodes(C5.large). Recently I am experiencing frequently that my domain gets into yellow state for some time around 30 mins and then i do nothing it changes to green.
  • when i used the query GET /_cluster/allocation/explain?pretty this is what i see
{
  "index" : "lgst-",
  "shard" : 4,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2021-01-06T13:15:38.721Z",
    "details" : "node_left [**************]",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "yes",
  "allocate_explanation" : "can allocate the shard",
  "target_node" : {
    "id" : "****************",
    "name" : "********************"
  },
  • I couldn't understand what does it mean and how do i over come it. Any help please would be appreciated.

Solution

  • Looks like you are using spot instances in your cluster and cause for this is that nodes in your AWS is not stable as shown clearly in the unassigned_info

    "unassigned_info" : {
        "reason" : "NODE_LEFT",
        "at" : "2021-01-06T13:15:38.721Z",
        "details" : "node_left [**************]",
        "last_allocation_status" : "no_attempt"
      },
    

    I would suggest chaning the instance types if you are using ec2 spot instances and check why nodes are getting disconnected in your cluster with AWS support.