Logstash parsing json

Question

I am trying to use log-stash to read input file 1.log in JSON format and write on elasticsearch. This is my log file:

{"key":"value00"}
{"key":"value01"}
{"key1":[{"key2":"value02"},{"key3":"value03"},{"key4":[{"key5":"value 04"}]}]}

and this is my configuration file:

input {
  file {
    type => "json"
    path => "/logstash/1.log"
  }
}
filter{
  json {
    source => "message"
    remove_field => ["message"]
  }
}
output {
    elasticsearch {
        hosts => ["192.168.1.6:9200"]
        user => "elastic"
        password => "something"
    }
}

the log-stash behaviour is completely random. Some times it works correctly but, some times it returns the following error for the same input structure:

Error parsing json {:source=>"message", :raw=>"4\"}]}]}", :exception=>#<LogStash::Json::ParserError: Unexpected character ('"' (code 34)): Expected space separating root-level values

Answer 1

My suggestion:

do not remove the message from the beginning during debugging, as you do not know what the source was that is having the issue. Do this conditionally based on the json filter outcome and if it failed write it to a file to figure out when it exactly fails.

From my experience it is mostly input that is eitehr wrong or was not expected to come in in that format.

Example based on your config:

input {
  file {
    type => "json"
    path => "/logstash/1.log"
  }
}
filter{
  json {
    source => "message"
  }
  if "_jsonparsefailure" not in [tags] {
    mutate {
      remove_field => ["message"]
    }
  }
}
output {
  if "_jsonparsefailure" not in [tags] {
    elasticsearch {
        hosts => ["192.168.1.6:9200"]
        user => "elastic"
        password => "something"
    }
  }
  if "_jsonparsefailure" in [tags] {
    file {
        path => "/write/to/this/file.txt"
    }
  }
}