Search code examples
elasticsearchindexinglogstash

Step-by-step guide for creating index?


I'm looking for a guide to create a index in elasticsearch, but it is not as simple as the guide given at:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

It seems pretty simple what I want to do but I just can't seem to get it working. Now, I want my index to be daily indices (same as the default logstash index) but with some changes. These changes includes a name change and a specific mapping for fields which have specific types. Now I know that I have to specify in the output-elasticsearch section in the logstash configuration that:

index => "name-%{+YYYY.MM.dd}"

The only information I found was that an index can be created based on a template and I tried creating the template but still nothing happens.

Creating the template I used the following:

PUT _template/ids
{
"template": "ids-*", 
"order":    0, 
"settings": {
"index": {
  "number_of_shards": 5,
  "number_of_replicas": 1
},
"mappings": {
  "log": {
    "_all": {
      "enabled": true,
      "omit_norms": true
    },
    "properties": {
      "@timestamp": {
        "type": "date",
        "format": "strict_date_optional_time||epoch_millis"
      },
      "@version": {
        "type": "string",
        "index": "not_analyzed"
      },
      "field1": {
        "type": "string",
        "index": "not_analyzed"
      },
      "field2": {
        "type": "string",
        "index": "not_analyzed"
      },

Solution

  • For daily indices with "some changes" it is nice to use templates.

    To check which templates are already set in the cluster use:

    GET {es_url}/_template
    

    To set new template to the cluster use:

    PUT {es_url}/_template/ids
    {
    "template": "ids-*", 
    "order":    0, 
    "settings": {
    "index": {
      "number_of_shards": 5,
      "number_of_replicas": 1
    },
    "mappings": {
      "log": {
        "_all": {
          "enabled": true,
          "omit_norms": true
        },
        "properties": {
          "@timestamp": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "@version": {
            "type": "string",
            "index": "not_analyzed"
          },
          "field1": {
            "type": "string",
            "index": "not_analyzed"
          },
          "field2": {
            "type": "string",
            "index": "not_analyzed"
          }
        }
      }
    }}}
    

    To delete an exsisting template use:

    DELETE {es_url}/_template/{template_name}
    

    If you set the "ids" template to the cluster- any document that will be inserted to the cluster, to index with name that matches "ids-*" (aka "ids-123", "ids-sheker", "ids-2016.05.02") will get the mapping of the inserted ids template.