Search code examples
elasticsearchelasticsearch-dsl

Performance of Terms Query with many elements

I'm planning to use a Terms Query with many terms (depending on the case up to 40-50k terms) in all my queries.

These terms will be fetched from another index using lookup as explained here. Elasticsearch takes them internally, so at least they won't go through the wire, but the query itself looks quite heavy.

I was wondering if the query performance will be fine. Anyway I'm planning to do a stress test, but not sure if this is going to escalate well. Someone had experience with these kind of queries or knows how Elasticsearch deals with them internally?

Thank you!

Solution

  • I opened an issue in the Elasticsearch repo about this matter, and as I feared, even using lookup, if used with many terms, this kind of query gets very slow.

    Also, I mentioned it in the issue, but I stress tested it and checked it myself:

    filtering with around 20 thousand terms make the query quite slow (more than 500ms).