Logfile won't apear in elasticsearch

Question

I'm very new to logstash and elasticsearch, I am trying to stash my first log to logstash in a way that I can (correct me if it is not the purpose) search it using elasticsearch....

I have a log that looks like this basically:

2016-12-18 10:16:55,404 - INFO - flowManager.py - loading metadata xml

So, I have created a config file test.conf that looks like this:

input {
  file {
    path => "/home/usr/tmp/logs/mylog.log"
    type => "test-type"
    id => "NEWTRY"
 }
}
filter {
  grok {
    match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} - %{LOGLEVEL:level} - %{WORD:scriptName}.%{WORD:scriptEND} - " }
  }
}
output {
  elasticsearch {
    hosts =>  ["localhost:9200"]
    index => "ecommerce"
    codec => line { format => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second} - %{level} - %{scriptName}.%{scriptEND} - \"%{message}\"" }
  }
}

And then : ./bin/logstash -f test.conf

I do not see the log in elastic search when I go to: http://localhost:9200/ecommerce OR to http://localhost:9200/ecommerce/test-type/NEWTRY

Please tell me what am I doing wrong.... :\

Thanks, Heather

Answer 1

I found a solution eventually-

I added both sincedb_path=>"/dev/null" (which from what I understood is for testing enviorment only) and start_position => "beginning" to the output file plugin and the file appeared both in elastic and in kibana

Thanks anyway for responding and trying to help!