How to automate the creation of elasticsearch index patterns for all days?

Question

I am using cloudwatch subscription filter which automatically sends logs to elasticsearch aws and then I use Kibana from there. The issue is that everyday cloudwatch creates a new indice due to which I have to manually create the new index pattern each day in kibana. Accordingly I will have to create new monitors and alerts in kibana as well each day. I have to automate this somehow. Also if there is better option with which I can go forward would be great. I know datadog is one good option.

Answer 1

Typical work flow will look like this (there are other methods)

• Choose a pattern when creating an index. Like staff-202001, staff-202002, etc
• Add each index to an alias. Like staff

This can be achieved in multiple ways, easiest is to create a template with index pattern , alias and mapping. Example: Any new index created matching the pattern staff-* will be assigned with given mapping and attached to alias staff and we can query staff instead of individual indexes and setup alerts.

We can use cwl--aws-containerinsights-eks-cluster-for-test-host to run queries.

POST _template/cwl--aws-containerinsights-eks-cluster-for-test-host
{
  "index_patterns": [
    "cwl--aws-containerinsights-eks-cluster-for-test-host-*"
  ],
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "firstName": {
        "type": "text"
      },
      "lastName": {
        "type": "text"
      }
    }
  },
  "aliases": {
    "cwl--aws-containerinsights-eks-cluster-for-test-host": {}
  }
}

Note: If unsure of mapping, we can remove mapping section.