Index Cleanup and Retention Elastic Cloud

Question

I am having a hard time finding a clear cut answer: What is the default retention time before deletion for an index in Elastic Cloud (ELK Stack) and can it be easily modified? We have recently migrated from a on-prem ELK stack to a cloud solution. Previously we did this with Curator.

Last post I found was from 2017 and said that this was not supported in the Cloud: https://discuss.elastic.co/t/configure-retention-period-for-different-index/106491 Has this changed?

Answer 1

I solved this by creating rollovers through the use of Index Lifecycle Managment (ILM), and Index Templates. Most information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html

However the gist of it is:

Create a ILM

Create an Index Template with a rollover alias:

PUT _template/example-template-name
{
  "index_patterns": [
    "example-index-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "name-of-lifecycle-created-earlier",
        "rollover_alias": "example-index-name"
      }
    }
  }
}

Then create the first index that has the rollover alias alias. Personally I use the rollover date system, with URL encoding:

PUT /%3Cexample-index-name-%7Bnow%2Fd%7D-1%3E
{
  "aliases": {
    "example-index-name": {
      "is_write_index" : true
    }
  }
}

After this simply write to the alias, not the index itself. It will automatically assign the correct index and rollover when the ILM limits are met.