Why does Elasticsearch allow me to index documents in a single node cluster that fails to meet quorum requirements?

Question

From https://www.elastic.co/guide/en/elasticsearch/guide/2.x/distrib-write.html:

Note that the number_of_replicas is the number of replicas specified in the index settings, not the number of replicas that are currently active. If you have specified that an index should have three replicas, a quorum would be as follows:

int( (primary + 3 replicas) / 2 ) + 1 = 3

But if you start only two nodes, there will be insufficient active shard copies to satisfy the quorum, and you will be unable to index or delete any documents.

I ran the following commands on a single-node cluster and I was able to index a document successfully even though the math above says I should not be able to index documents.

curl -X DELETE http://localhost:9200/a/?pretty
curl -X PUT -siH 'Content-Type: application/json' \
     http://localhost:9200/a?pretty -d '{

    "settings": {
        "number_of_replicas": 3
    }
}'
curl -sH 'Content-Type: application/json' -X PUT http://localhost:9200/a/a/1?pretty -d '{"a": "a"}'
curl -si http://localhost:9200/_cluster/health?pretty
curl -si http://localhost:9200/a/a/1?pretty

Here is the output:

$ curl -X PUT -siH 'Content-Type: application/json' \
     http://localhost:9200/a?pretty -d '{

    "settings": {
        "number_of_replicas": 3
    }
}'
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 77

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "a"
}

$ curl -sH 'Content-Type: application/json' -X PUT http://localhost:9200/a/a/1?pretty -d '{"a": "a"}'
{
  "_index" : "a",
  "_type" : "a",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 4,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

$ curl -si http://localhost:9200/_cluster/health?pretty
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 468

{
  "cluster_name" : "docker-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 15,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 25.0
}

$ curl -si http://localhost:9200/a/a/1?pretty
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 124

{
  "_index" : "a",
  "_type" : "a",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "a" : "a"
  }
}

How was I able to index a document with just 1 node even though I have configured the index to have 3 replica shards and the math says I must have at least 3 nodes?

Answer 1

That was an old quorum rule in ES 2.x. As of ES 5.x the write concistency checks have been changed a little bit and a yellow cluster, i.e. a cluster will all primary shards allocated, will pass the consistency checks for write operations and allow you to index and delete documents.

Now the way to decide whether writes can be made is by using the wait_for_active_shards parameter in the indexing operation. By default, if all primary shards are up the index operation will be allowed. You can override that settings by specifying the number of shards that you wish to be active before an indexing operation should be authorized, e.g. wait_for_active_shards=all which is equivalent to wait_for_active_shards=4 (4 = 1 primary + 3 replicas) in your case. If you want the same quorum rule as before you'd specify wait_for_active_shards=3.

More info can be found in the official documentation and here