Search code examples
elasticsearchelastic-stack

ELK host.ip has conflicting data types in the kong-* index pattern

I am using ELK version 8.10.x for Kong API Gateway logs. In Kibana dashboard I found that host.ip has conflict data type. After investigation, I found that there is one index call kong-01 has host.ip field as text type while the other indices is ip type.

This conflict causing issues when visualizing the data.

How to modify the data type for host.ip field to be ip type for the existing index please?

This is the current mappings configuration for kong-01 index:

{
  "kong-01": {
    "mappings": {
      "host.ip": {
        "full_name": "host.ip",
        "mapping": {
          "ip": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

I tried to create a new index using the below command:

PUT kong-01-new
{
  "mappings": {
    "host.ip": {
      "full_name": "host.ip",
      "mapping": {
        "ip": {
          "type": "ip"
        }
      }
    }
  }
}

But I have got the below errors:

{

  "error": {

    "root_cause": [

      {

        "type": "mapper_parsing_exception",

        "reason": "Root mapping definition has unsupported parameters:  [host.ip : {mapping={ip={type=ip}}, full_name=host.ip}]"

      }

    ],

    "type": "mapper_parsing_exception",

    "reason": "Failed to parse mapping: Root mapping definition has unsupported parameters:  [host.ip : {mapping={ip={type=ip}}, full_name=host.ip}]",

    "caused_by": {

      "type": "mapper_parsing_exception",

      "reason": "Root mapping definition has unsupported parameters:  [host.ip : {mapping={ip={type=ip}}, full_name=host.ip}]"

    }

  },

  "status": 400

}

After Re-index completed, I have got the below error for kong01-new index:

Index lifecycle error

illegal_argument_exception: index.lifecycle.rollover_alias [kong] does not point to index [kong-01-new]

I re-indexed again using the same naming convention of ilm_pattern-> "{now/d}-000001" configured in logstash for the new index but still getting the below error:

Index lifecycle error

illegal_argument_exception: index.lifecycle.rollover_alias [kong] does not point to index [kong-2022-11-24-000001]
Solution

  • You can perform reindex and remove the old index with wrong mapping. Here are the steps:

    UPDATED - Create a new index with the proper mapping

    PUT kong-01-new
{
  "mappings": {
    "properties": {
      "host": {
        "properties": {
          "ip": {
            "type": "ip"
          }
        }
      }
    }
  }
}

    Run the reindex

    POST _reindex?wait_for_completion=false
{
  "conflicts": "proceed",
  "source": {
    "index": "kong-01"
  },
  "dest": {
    "index": "kong-01-new"
  }
}

    Copy the task_id and wait until the reindex completed

    GET _tasks/{task_id from the output of reindex API}

    Remove the old index and add old index alias to the new index

    POST _aliases
{
  "actions": [
    {
      "remove_index":
      {
        "index": "kong-01"
      }
    },
    {
      "add": {
        "index": "kong-01-new",
        "alias": "kong-01"
      }
    }
  ]
}

    Image added to show the alias. enter image description here