Input data from CSV file to logstash

Question

I have a csv file, with the following headers:

"PacketId","MACAddress","Date","PacketLength","SourceIP","SourcePort","DestIP","DestPort"

I want to index the data to ElasticSearch using LogStash, and not able to write the filter for the same.

filter {
    grok {
        match => message => "%{IP:SourceIP}"
    }
    }

The above filter gives a nice extraction of SourceIP field, but how do I write the grok pattern to extract it for all fields.

Answer 1

Let the following CSV file:

1,00-14-22-01-23-45,13/09/2015,32,128.248.1.43,9980,128.248.23.13,9880
1,01-74-02-84-13-98,14/09/2015,64,128.248.1.94,9280,128.248.13.84,9380

Here Logstash configuration you must set:

input {
    file {
        path => "/path/of/your/csv/test.csv"
        sincedb_path => "/path/of/your/csv/test.idx"
        start_position => "beginning"
    }
}

filter {
    csv {
        separator => ","
        columns => ["PacketId","MACAddress","Date","PacketLength","SourceIP","SourcePort","DestIP","DestPort"]
    }
}

output {
    stdout {
        codec => rubydebug      
    }
}

You will get in output result:

{
         "message" => [
        [0] "1,00-14-22-01-23-45,13/09/2015,32,128.248.1.43,9980,128.248.23.13,9880"
    ],
        "@version" => "1",
      "@timestamp" => "2015-09-14T20:11:28.976Z",
            "host" => "MyHost.local",
            "path" => "/path/of/your/csv/test.csv",
        "PacketId" => "1",
      "MACAddress" => "00-14-22-01-23-45",
            "Date" => "13/09/2015",
    "PacketLength" => "32",
        "SourceIP" => "128.248.1.43",
      "SourcePort" => "9980",
          "DestIP" => "128.248.23.13",
        "DestPort" => "9880"
}
{
         "message" => [
        [0] "1,01-74-02-84-13-98,14/09/2015,64,128.248.1.94,9280,128.248.13.84,9380"
    ],
        "@version" => "1",
      "@timestamp" => "2015-09-14T20:11:28.978Z",
            "host" => "MyHost.local",
            "path" => "/path/of/your/csv/test.csv",
        "PacketId" => "1",
      "MACAddress" => "01-74-02-84-13-98",
            "Date" => "14/09/2015",
    "PacketLength" => "64",
        "SourceIP" => "128.248.1.94",
      "SourcePort" => "9280",
          "DestIP" => "128.248.13.84",
        "DestPort" => "9380"
}

Regards, Alain