Spring-data-elasticsearch: Result window is too large (index.max_result_window)

Question

We retrieve information from Elasticsearch 2.7.0 and we allow the user to go through the results. When the user requests a high page number we get the following error message:

Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter

The thing is we use pagination in our requests so I don't see why we get this error:

@Autowired
private ElasticsearchOperations elasticsearchTemplate;
...
elasticsearchTemplate.queryForPage(buildQuery(query, pageable), Document.class);
...
private NativeSearchQuery buildQuery() {
    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    boolQueryBuilder.should(QueryBuilders.boolQuery().must(QueryBuilders.termQuery(term, query.toUpperCase())));
    NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder().withIndices(DOC_INDICE_NAME)
                .withTypes(indexType)
                .withQuery(boolQueryBuilder)
                .withPageable(pageable);
    return nativeSearchQueryBuilder.build();
}

I don't understand the error because we retreive pageable.size (20 elements) everytime... Do you have any idea why we get this?

Answer 1

Unfortunately, Spring data elasticsearch even when paging results searchs for a much larger result window in the elasticsearch. So you have two options, the first is to change the value of this parameter.

The second is to use the scan / scroll API, however, as far as I understand, in this case the pagination is done manually, as it is used for infinite sequential reading (like scrolling your mouse). A sample:

List<Pessoa> allItens = new ArrayList<>();
    String scrollId = elasticsearchTemplate.scan(build, 1000, false, Pessoa.class);
    Page<Pessoa> page = elasticsearchTemplate.scroll(scrollId, 5000L, Pessoa.class);
    while (true) {
        if (!page.hasContent()) {
            break;
        }
        allItens.addAll(page.getContent());
        page = elasticsearchTemplate.scroll(scrollId, 5000L, Pessoa.class);
    }

This code, shows you how to read ALL the data from your index, you have to get the requested page inside scrolling.