Python to elastic search to query logs in different timeframes

Question

We have Elastic search cluster with thousands of applications. We would like to read logs through python script from elastic cluster.

Use Cases :

1. Get logs of last 1 hour or 1 day
2. If it has error 500 or 404

We would like to get this kind of details from cluster.

Able to connect to cluster and in body provided query with match condition.

But I was struck to provide filter for errors from logs and getting last 1h data

Answer 1

from datetime import datetime, timedelta
from elasticsearch import Elasticsearch

es = Elasticsearch(['your_elasticsearch_host'], port=9200)

index_pattern = "your_index_pattern*"


query = {
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h"  # Logs from the last 1 hour
            }
          }
        },
        {
          "terms": {
            "status_code": [500, 404]  # Filter logs based on status code 500 or 404
          }
        }
      ]
    }
  }
}


results = es.search(index=index_pattern, body=query)

//response generation.
for hit in results['hits']['hits']:
    print(hit['_source'])

filter logs from the last 1 hour ("gte": "now-1h")

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html