Search code examples
elasticsearchelasticsearch-aggregation

ElasticSearch - Search between a range of dates to compare them


I am new to ElasticSearch (using version 7.6) and trying to find out how to search between two periods in time. One query I'm trying out is to query week-12 of 2019 and week-12 of 2020. The idea is to compare the results. While reading the documentation and searching for samples I have came close to what I'm looking for.

The easy way was to fire two queries with both different dates. But I would like to limit the amount of queries. The latest query I have written based on reading the docs is with the use of aggregations, but I'm not sure this is the right way:

GET sample-data_*/_search/
{
"query": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": "2020-03-20 08:00:00",
            "lte": "2020-03-27 08:00:00"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "range": {
      "date_range": {
        "field": "date",
        "format": "8yyyy-MM-dd",
        "ranges": [
          {
            "from": "2019-03-20",
            "to": "2019-03-27",
            "key": "last_years_week" 
          },
          {
            "from": "2020-03-20",
            "to": "2020-03-27",
            "key": "this_years_week"
          }
        ],
        "keyed": true
      }
    }
  }
}

The results are coming in followed by the aggregations, but they do not contain the data that I am looking for. One of the results are returned:

{
    "_index" : "sample-data_2020_03_26",
    "_type" : "_doc",
    "_id" : "JyhcfWFFz0s1vwizjgxh",
    "_score" : 1.0,
    "_source" : {
        "@timestamp" : "2020-03-26 00:00:00",
        "name" : "TEST0001",
        "count" : "150",
        "total" : 3000
    }
}
...
"aggregations" : {
    "range" : {
      "buckets" : {
        "last_years_week" : {
          "from" : 1.55304E12,
          "from_as_string" : "2019-03-20",
          "to" : 1.5536448E12,
          "to_as_string" : "2019-03-27",
          "doc_count" : 0
        },
        "this_years_week" : {
          "from" : 1.5846624E12,
          "from_as_string" : "2020-03-20",
          "to" : 1.5852672E12,
          "to_as_string" : "2020-03-27",
          "doc_count" : 0
        }
      }
    }
  }

My question is: what could be an efficient way to query data between two dates of different years using ElasticSearch, so they could be used to compare the numbers?

I would be happy to read more about the, for me complex, ElasticSearch query if you could point me into the right direction.

Thank you!


Solution

  • Not posting the working solution with the Elasticsearch query but as discussed in the question comments, summarize it in the form of the answer which some useful links.

    Range queries on date fields are very useful to quickly search between date ranges, also supports various math operations on date fields.

    aggregation on date range will also be useful and The main difference between this aggregation and the normal range aggregation is that the from and to values can be expressed in Date Math expression useful if you want to have aggregations on your date range and it supports data math format as mention below: