Query all in ElasticSearch using Nest v. 2.1

Question

var settings = new ConnectionSettings(Constants.ElasticSearch.Node);
var client = new ElasticClient(settings);

var response = client.Search<DtoTypes.Customer.SearchResult>(s =>
    s.From(0)
    .Size(100000)
    .Query(q => q.MatchAll()));

It works when the size is smaller, but I want to retrieve all documents in an index that has over 100k documents. Must be a configuration setting I'm missing to get around a limit. I've also tried Take() instead of Size()

The Debug Info returned back is

"Invalid NEST response built from a unsuccesful low level call on POST: /_search\r\n# Audit trail of this API call:\r\n - BadResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.2964038\r\n# ServerError: ServerError: 500Type: search_phase_execution_exception Reason: \"all shards failed\"\r\n# OriginalException: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.\r\n at System.Net.HttpWebRequest.GetResponse()\r\n at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\users\russ\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 138\r\n# Request:\r\n\r\n# Response:\r\n\r\n"

Answer 1

Elasticsearch has a soft limit on the amount of results it allows to return. If you want more then 10.000 results in one go, you should use the scan and scroll functionality :)

From the Elasticsearch documentation:

"Note that from + size can not be more than the index.max_result_window index setting which defaults to 10,000. See the Scroll API for more efficient ways to do deep scrolling."

Reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html https://nest.azurewebsites.net/nest/search/scroll.html