Logstash Auto update Data

Question

I am using the latest version of logstash(7.6.2). I tried uploading a sample data and was able to successfully upload it into the elasticsearch using logstash(enabled auto-reload) and was able to see the index in the Kibana interface.

But, when I make changes to the below config file, I was unable to see the updated data in the Kibana interface. I was trying to remove the mutate filter plugin and the logstash pipeline reloaded but the data in Kibana is not updated. Interestingly it didn't throw up any errors.

Sample.conf

input{
file{
    path => "/usr/local/Cellar/sample.log"
    start_position => "beginning"
}
}

filter{
    grok{
        match => ["message", "%{TIMESTAMP_ISO8601:timestamp_string}%{SPACE}%{GREEDYDATA:line}"]
    }
    date{
        match => ["timestamp_string", "ISO8601"]
    }
    mutate{
        remove_field => [message, timestamp_string]
    }
}
output{
elasticsearch{
    hosts => ["localhost:9200"]
    index => "sample"
}
stdout{
    codec => rubydebug
}
}

Any help here is appreciated. TIA

P.S. - I am new to ElasticSearch!

Answer 1

If you want to parse again a complete file, you need to :

• delete sindedb files
• OR only delete the corresponding line in sincedb file

Then, restart Logstash. Logstash will reparse the file.

For more info: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#sincedb_path