Search code examples
elasticsearchkibana

Elastic Search cluster setup giving inconsistent search results while using pagination

I've an Elastic Search(version 6.8.23) cluster of 2 nodes, 1 primary shard & 1 replica shard. Initially loaded with a few, around 1500, records into it.

Some details about the cluster are below:

Health Status: Green

Output of POST /_cat/shards/<index-name>?v

index    shard prirep state   docs store ip           node
index-6  0     p      STARTED 1546 6.5mb ***.***.**.1 node-1
index-6  0     r      STARTED 1546 6.5mb ***.***.**.2 node-2

Output of POST /_cat/segments/<index-name>?v

index    shard prirep ip           segment generation docs.count docs.deleted    size size.memory committed searchable version compound
index-6  0     p      ***.***.**.1 _4               4        154            4   1.9mb        8372 true      true       7.7.3   true
index-6  0     p      ***.***.**.1 _b              11       1303           78   3.5mb       38020 true      true       7.7.3   false
index-6  0     p      ***.***.**.1 _c              12         82            0 661.7kb       10525 true      true       7.7.3   true
index-6  0     p      ***.***.**.1 _d              13          6            0 367.4kb        4359 true      true       7.7.3   true
index-6  0     p      ***.***.**.1 _e              14          1            0   4.3kb        1635 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _6               6        173            7     2mb        8357 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _b              11         26            1  48.1kb        7229 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _c              12        994           67   2.8mb       30852 true      true       7.7.3   false
index-6  0     r      ***.***.**.2 _d              13        264            7 565.7kb        7866 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _e              14         82            0 661.7kb       10525 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _f              15          6            0 367.4kb        4359 true      true       7.7.3   true
index-6  0     r      ***.***.**.2 _g              16          1            0   4.3kb        1635 true      true       7.7.3   true

Note: compound field false for couple of segments.

In this setup, if we hit the following search request multiple times, it give 2 different result sets(seems like different for each node/shard).

POST /index-6/_search?from=0&size=10

The results are not consistent every time.

Few questions:

  1. Why it is giving inconsistent results?
  2. Why the compound value is false for some segments?
Solution

  • If you want to paginate over your index, you also need to specify a sort value to make sure the pagination order is deterministic.

    POST /index-6/_search?from=0&size=10&sort=id:asc
                                        ^
                                        |
                                   add sorting

    Note that if your index is actively being written to, the order might change. If this is the case, in 6.8 you might want to resort to using the Scroll API in order to freeze the search context for the duration of the scroll and return consistent results.

    Regarding your second question, the compound boolean simply means that Lucene merged all files from the segment into a single file to save file descriptors (if true).