Why is ElasticSearch request failing after 10.000 documents?

Question

We are using ElasticSearch.NET / NEST to query an ElasticSearch configuration. The plan is to fetch in batches of 1000 documents and process them before fetching the next 1000 documents

However it always fails after processing 10 batches

ELK Search failed Invalid NEST response built from a unsuccessful (500)

If we change the batch-size to 10,000 it will fail after 1 batch With batch size of 100 it will fail after 100 batches Failure is always efter 10.000 documents

The code looks something like this

        private void ProcessRequest(SearchRequest request)
    {
        request.Size = 1000;
        for (request.From = 0; request.From < 1_000_000; request.From += request.Size)
        {
            Console.WriteLine(request.From);
            var responses = _client.Search<GroupStaticElkDocument>(request);
            foreach (var response in responses)
                _requestCounter.Add(response.ToRequest());
        }
    }

Answer 1

Maybe you shuld try paging/scrolling