logstash output elasticsearch index with sequence number

Question

I am using AWS Elastic Search (Version 7.10) with Logstash 7.10. The intention is to send the content from logstash to elastic search and rollover the index after the particular size or time using policy.

policy: {
    "policy_id": "Rollover_Policy",
    "description": "roller index",
    "last_updated_time": 1634910129219,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "1mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm"
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 1
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "1h"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "products*"
            ],
            "priority": 100,
            "last_updated_time": 1634910129219
        }
    ]
}

While I am trying to set ilm_enabled to true in logstash output plugin, it is not able to connect with elastic search xpack API.

Note : xpack and ILM are not supported in AWS elastic search.

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => true
        ilm_rollover_alias => "products"
        ilm_pattern => "{now/d}-000001"
        ilm_policy => "Rollover_Policy"
}

So I have changed ilm_enabled flag to false and tried below option

elasticsearch {
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false
        index => "products-%{+YYYY.MM.dd}-000001"
}

Now the problem is that even after the rollover, logstash is still sending the documents to 001 index instead of new index. If I don't give -000001 in index name, then rollover is getting failed.

Answer 1

Create an index with following REST request in elastic. Since the index name is having date pattern, the rollover will create new index with current date.

PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E
{
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":1
  },
  "aliases": {
    "products":  {
      "is_write_index": true
    }
  }

Create a template for index pattern along with rollover alias

PUT _index_template/products_logs
{
  "index_patterns": [
    "products*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "opendistro": {
        "index_state_management": {
          "rollover_alias": "products"
        }
      }
    }
  }
}

In logstash output plugin give the below details to send the data to elastic search

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false 
        index => "products"
}

Note : the index name represents alias name of the index.