Logstash and Elastic upgrade

Question

I had a functional Logstash and Elasticsearch on version 5.1.

I deleted all indices, then upgraded to 6.1.

Now, when Logstash receives some event from Filebeat (Which stills version 5.1), it throws this error:

[2017-12-27T17:29:16,463][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch.
{
  :status => 400,
  :action => ["index", {:_id=>nil, :_index=>"logstash-2017.12.27", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x34de85bd>],
  :response => {
    "index" => {
      "_index" => "logstash-2017.12.27",
      "_type" => "doc",
      "_id" => nil,
      "status" => 400,
      "error" => {
        "type" => "mapper_parsing_exception",
        "reason" => "Failed to parse mapping [_default_]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.",
        "caused_by" => {
          "type" => "mapper_parsing_exception",
          "reason" => "[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."
        }
      }
    }
  }
}

I have even tried using an extremely simplistic pipeline, as you can see here:

input {
  beats {
    port => 5044
  }
}   

filter {
    json {
      source => "message"
    }   
}   

output {
  elasticsearch { hosts => ["localhost:9200"] }
}

Yet it throws this error over and over.

Any idea what can be wrong here?

Answer 1

take a look at changes in mapping, introduced in elasticsearch 6.0

you need to remove the include_in_all mapping parameter from your index template.

can you paste here your template/mapping?