I am having a hard time finding a clear cut answer: What is the default retention time before deletion for an index in Elastic Cloud (ELK Stack) and can it be easily modified? We have recently migrated from a on-prem ELK stack to a cloud solution. Previously we did this with Curator.
Last post I found was from 2017 and said that this was not supported in the Cloud: https://discuss.elastic.co/t/configure-retention-period-for-different-index/106491 Has this changed?
I solved this by creating rollovers through the use of Index Lifecycle Managment (ILM), and Index Templates. Most information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html
However the gist of it is:
Create a ILM
Create an Index Template with a rollover alias:
PUT _template/example-template-name
{
"index_patterns": [
"example-index-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "name-of-lifecycle-created-earlier",
"rollover_alias": "example-index-name"
}
}
}
}
Then create the first index that has the rollover alias alias. Personally I use the rollover date system, with URL encoding:
PUT /%3Cexample-index-name-%7Bnow%2Fd%7D-1%3E
{
"aliases": {
"example-index-name": {
"is_write_index" : true
}
}
}
After this simply write to the alias, not the index itself. It will automatically assign the correct index and rollover when the ILM limits are met.