Search code examples
elasticsearchdocker-composelogstash

How to input a JSON file (array form) in Logstash?


I tried many things in order to input my JSON file (array form), I don't have any results so far

input{
    file{
        path=> "/usr/share/logs/Docs.json"


        #codec=> "json" tried
        #codec=> json {charset => "ISO-8859-1"} tried
        #codec => json{ } tried


        start_position=> "beginning"
        sincedb_path=> "/dev/null"
    }
}
filter{
    json{
        source=> "message"
    }
 }
output{
    stdout{
        codec=> "json"
    }
    
    elasticsearch{
        hosts=> ["http://es1:9200"]
        index=> "index_abc"
    }
}

JSON file format (all on the same line) :

[{"id":1,"something":"text1"},{"id":2,"something":"text2"},{"id":3,"something":"text3"}]

If you could me I would appreciate it very much.


Solution

  • Problem solved, it was because of the encoding, I used the jq utility in order to transform my JSON file to the right format (for Logstash), which is :

    {"id":1,"something":"text1"}
    {"id":2,"something":"text2"}
    {"id":3,"something":"text3"}
    

    and I didn't notice at first but jq changed the encoding of my file during the transformation, it ended up in UTF-16 LE. So, I changed (and saved) the encoding to UTF-8 with VS Code, and it worked fine afterwards.

    My updated code that works now :

    input{
        file{
            path=> "/usr/share/logs/Docs_formatted.json"
            codec=> "json"
            start_position=> "beginning"
            sincedb_path=> "/dev/null"
        }
    }
    filter{}
    output{
        stdout{}
        
        elasticsearch{
            hosts=> ["http://es1:9200"]
            index=> "index_abc"
        }
    }
    

    For those who are interested, I used this command line in order to transform my JSON file to the right format (Windows command line):

    type Docs.json | jq -c '.[]' > Docs_formatted.json