Logstash - Bad event type, Non-string/integer type value set

Question

I want to send logs from Java app using Log4j to my logstash server over TCP. Communication between two servers works great but When I receive logs from my Java app, I have an error into logstash's logs file:

[2017-04-19T09:15:18,549][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"hadoopws_log-2017.04.19", :_type=>"[\"log4j\", \"001\"]", :_routing=>nil}, 2017-04-19T07:15:18.486Z 10.10.2.100:60283 "ROUM" 8962 001 2 3136 109 3245], :response=>{"index"=>{"_index"=>"hadoopws_log-2017.04.19", "_type"=>"[\"log4j\", \"001\"]", "_id"=>"AVuFEL1uRiFXDsNz9hxO", "status"=>400, "error"=>{"type"=>"invalid_type_name_exception", "reason"=>"mapping type name [[\"log4j\", \"001\"]] should not include ',' in it"}}}}

My logstash's config file:

input {
    log4j {
    mode => "server"
    host => "hadoopmasterdev"
    port => 3456
    type => "log4j"
  }
}

filter {
   grok {
     match => { "message" => "%{DATA:request} %{WORD:idgroup} %{WORD:type} %{NUMBER:nb} %{NUMBER:process_time} %{NUMBER:render_time} %{NUMBER:total_time}" }
   }
}

output {
  elasticsearch {
    hosts => ["hadoopmasterdev:9200"]
    index => "hadoopws_log-%{+YYYY.MM.dd}"
  }
}

Then, I send those logs:

"test 1" 28 001 26 33 116 149

When I try to validate my Grok pattern using GrokDebug, it works well...

Thank you so much :)

Answer 1

You need to change the third field name from typeto something else, because that interferes with the type: log4j field defined in your log4j input. As a result, the type field becomes an array that contains two values, namely log4j and 001 and that cannot be used to define the mapping type in the elasticsearch output.

Simply change your grok pattern to this and it will work.

grok {
   match => { "message" => "%{DATA:request} %{WORD:idgroup} %{WORD:datatype} %{NUMBER:nb} %{NUMBER:process_time} %{NUMBER:render_time} %{NUMBER:total_time}" }
}