ELK index name doesn't change on rollover on the next day

I am using ELK (elasticsearch-8.12.0-1.x86_64) to store kong API gateway logs. I am using ILM (Index Lifecycle Management) policy to manage the index retention and I mentioned it into logstash pipeline configuration file.

I noticed that the new created indices are created using the below naming convention although they have been created in different days:

kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-11-17-000004
kong-2022-11-17-000005
kong-2022-11-17-000006

How to change the naming convention to include the creation date like the following:

kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-12-25-000001
kong-2023-01-01-000001

/etc/logstash/kong.conf

elasticsearch {
    hosts => ["https://elastic01:elastic_port" , "https://elastic02:elastic_port" , "https://elastic03:elastic_port"]
    user => "elastic_user"
    password => elastic_user_password
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/http_ca.crt"
    ilm_rollover_alias => "kong"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "kong-index-policy"

kong-index-template

{
  "index": {
    "lifecycle": {
      "name": "kong-index-policy",
      "rollover_alias": "kong"
    },
    "mapping": {
      "total_fields": {
        "limit": "10000"
      }
    },
    "refresh_interval": "5s"
  }
}

kong-index-policy

{
  "policy": "kong-index-policy",
  "phase_definition": {
    "min_age": "0ms",
    "actions": {
      "rollover": {
        "max_age": "180d",
        "max_primary_shard_size": "10gb"
      },
      "set_priority": {
        "priority": 100
      }
    }
  },

I tried to configure ILM policy to manage the indices rollover and create the new index using the creation date but it is not working properly.

Update01: I tried the following command:

PUT %3Ckong-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "kong": {
     "is_write_index": true
   }
 }
}

But I have got the following error:

```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "alias [kong] has more than one write index [kong-2024.06.05-000001,kong-2022-11-24-000009]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "alias [kong] has more than one write index [kong-2024.06.05-000001,kong-2022-11-24-000009]"
  },
  "status": 500
}
```

To solve that error, I toggled kong-2022-11-24-000009 index with the following then proceeded with the provided solution:

```
POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "kong-2022-11-24-000009",
        "alias": "kong",
        "is_write_index": false
      }
    }]
}
```

Solution

If you use an index alias for time series data, you can use date math in the index name to track the rollover date. For example, you can create an alias that points to an index named <my-index-{now/d}-000001>. If you create the index on May 6, 2099, the index’s name is my-index-2099.05.06-000001. If you roll over the alias on May 7, 2099, the new index’s name is my-index-2099.05.07-000002.

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#roll-over-index-alias-with-write-index

PUT %3Cmy-index-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "my-alias": {
      "is_write_index": true
    }
  }
}
#response
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my-index-2024.06.05-000001"
}

POST my-alias/_rollover
#response
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": "my-index-2024.06.05-000001",
  "new_index": "my-index-2024.06.05-000002",
  "rolled_over": true,
  "dry_run": false,
  "conditions": {}
}

Note: You should create the first rollover index manually.