I am using ELK (elasticsearch-8.12.0-1.x86_64) to store kong API gateway logs. I am using ILM (Index Lifecycle Management) policy to manage the index retention and I mentioned it into logstash pipeline configuration file.
I noticed that the new created indices are created using the below naming convention although they have been created in different days:
kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-11-17-000004
kong-2022-11-17-000005
kong-2022-11-17-000006
How to change the naming convention to include the creation date like the following:
kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-12-25-000001
kong-2023-01-01-000001
/etc/logstash/kong.conf
elasticsearch {
hosts => ["https://elastic01:elastic_port" , "https://elastic02:elastic_port" , "https://elastic03:elastic_port"]
user => "elastic_user"
password => elastic_user_password
ssl => true
ssl_certificate_verification => false
cacert => "/etc/logstash/http_ca.crt"
ilm_rollover_alias => "kong"
ilm_pattern => "{now/d}-000001"
ilm_policy => "kong-index-policy"
kong-index-template
{
"index": {
"lifecycle": {
"name": "kong-index-policy",
"rollover_alias": "kong"
},
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s"
}
}
kong-index-policy
{
"policy": "kong-index-policy",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "180d",
"max_primary_shard_size": "10gb"
},
"set_priority": {
"priority": 100
}
}
},
I tried to configure ILM policy to manage the indices rollover and create the new index using the creation date but it is not working properly.
Update01: I tried the following command:
PUT %3Ckong-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"kong": {
"is_write_index": true
}
}
}
But I have got the following error:
```
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "alias [kong] has more than one write index [kong-2024.06.05-000001,kong-2022-11-24-000009]"
}
],
"type": "illegal_state_exception",
"reason": "alias [kong] has more than one write index [kong-2024.06.05-000001,kong-2022-11-24-000009]"
},
"status": 500
}
```
To solve that error, I toggled kong-2022-11-24-000009 index with the following then proceeded with the provided solution:
```
POST /_aliases
{
"actions": [
{
"add": {
"index": "kong-2022-11-24-000009",
"alias": "kong",
"is_write_index": false
}
}]
}
```
If you use an index alias for time series data, you can use date math
in the index name to track the rollover date. For example, you can
create an alias that points to an index named
<my-index-{now/d}-000001>
. If you create the index on May 6, 2099, the
index’s name is my-index-2099.05.06-000001
. If you roll over the alias
on May 7, 2099, the new index’s name is my-index-2099.05.07-000002
.
PUT %3Cmy-index-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"my-alias": {
"is_write_index": true
}
}
}
#response
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "my-index-2024.06.05-000001"
}
POST my-alias/_rollover
#response
{
"acknowledged": true,
"shards_acknowledged": true,
"old_index": "my-index-2024.06.05-000001",
"new_index": "my-index-2024.06.05-000002",
"rolled_over": true,
"dry_run": false,
"conditions": {}
}
Note: You should create the first rollover index manually.