Search code examples
elasticsearchlogstashkibana

logstash does not load data to elasticsearch index

I have created an index in elasticsearch to store some test logs, but I can't get the data to load from logstash. I create the index in ES:

    put sonicwall
    {
      "mappings":{
        "properties":{
          "Time": { "type":"text"},
          "Category": { "type":"text"},
          "Group": { "type":"text"},
          "Event":{ "type":"text"},
          "Priority":{ "type":"text"}
        }
      }
    }

The index is created enter image description here

Now, my logstash config file:

    input{
        file{
            path => "C:/Elastic/logsPrueba/log3.csv"
            start_position => beginning
        }
    }
    filter{
        csv{
            separator => ","
            columns => ["Time","Category","Group","Event","Priority"]
        }
        
    }
    output{
        
        elasticsearch{
            hosts => ["localhost:9200"]
            index => "sonicwall"
        }
        stdout {}
    }

This is the CSV

enter image description here

when I run logstash, the data is never loaded enter image description here

Can someone help me?

Solution

  • I think I have discovered the problem. When the csv file is read for the first time, some kind of record is saved at the end of the file and then logstash is not able to start from the beginning. I have tried to fix it with sincedb_path = "NULL" on Windows or sincedb_path="/dev/null" on Linux, but logstash is able to start reading the file from the beginning again and again. If I add a new log line and run logstash, only the newly added line is painted.