configuring the logstash-output-csv

Question

I am pretty new to logstash and I have been trying to convert an existing log into a csv format using the logstash-output-csv plugin.

My input log string looks as follows which is a custom log written in our application.

'128.111.111.11/cpu0/log:5988:W/"00601654e51a15472-76":687358:<9>2015/08/18 21:06:56.05: comp/45 55% of memory in use:  2787115008 bytes (change of 0)'

I wrote a quick regex and added it to the patterns_dir using the grok plugin. My pattern is as follows :

IP_ADDRESS [0-9,.]+
CPU [0-9]
NSFW \S+
NUMBER [0-9]
DATE [0-9,/]+\s+[0-9]+[:]+[0-9]+[:]+[0-9,.]+
TIME \S+
COMPONENT_ID \S+
LOG_MESSAGE .+

without adding any csv filters I was able to get this output.

{
       "message" => "128.111.111.11/cpu0/log:5988:W/"00601654e51a15472-76":687358:<9>2015/08/18 21:06:56.05: comp/45 55% of memory in use:  2787115008 bytes (change of 0)",
      "@version" => "1",
    "@timestamp" => "2015-08-18T21:06:56.05Z",
          "host" => "hostname",
          "path" => "/usr/phd/raveesh/sample.log_20150819000609",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

This is my configuration in order to get the csv as an output

input {
    file {
        path => "/usr/phd/raveesh/temporary.log_20150819000609"
        start_position => beginning
    }
}

filter {
    grok {
        patterns_dir => "./patterns"
        match =>["message", "%{IP_ADDRESS:ipaddress}/%{CPU:cpu}/%{NSFW:nsfw}<%{NUMBER:number}>%{DATE}:%{SPACE:space}%{COMPONENT_ID:componentId}%{SPACE:space}%{LOG_MESSAGE:logmessage}" ]
        break_on_match => false
    }
    csv {
        add_field =>{"ipaddress" => "%{ipaddress}" }
}
}

output {
  # Print each event to stdout.
  csv {
    fields => ["ipaddress"]
    path => "./logs/firmwareEvents.log"
}
  stdout {
    # Enabling 'rubydebug' codec on the stdout output will make logstash
    # pretty-print the entire event as something similar to a JSON representation.
    codec => rubydebug
  }
}

The above configuration does not seem to give the output. I am trying only to print the ipaddress in a csv file but finally I need to print all the captured patterns in a csv file. so I need the output as follows :

128.111.111.111,cpu0,nsfw, ....

Could you please let me know the changes i need to make. ?

Thanks in advance

EDIT:

I fixed the regex as suggested using the tool http://grokconstructor.appspot.com/do/match#result

Now my regex filter looks as follows :

%{IP:client}\/%{WORD:cpu}\/%{NOTSPACE:nsfw}<%{NUMBER:number}>%{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}%{SPACE:space}%{TIME:time}:%{SPACE:space2}%{NOTSPACE:comp}%{SPACE:space3}%{GREEDYDATA:messagetext}

How do I capture the individual splits and save it as a csv ?

Thanks

EDIT:

I finally resolved this using the File plugin . output {

file{ path => "./logs/sample.log" message_pattern =>"%{client},%{number}" } }

Answer 1

The csv tag in the filter section is for parsing the input and exploding the message to key/value pairs.

In your case you are already parsing the input with the grok, so I bet you don't need the csv filter.

But in the output we can see there is a gorkfailure

{
       "message" => "128.111.111.11/cpu0/log:5988:W/"00601654e51a15472-76":687358:<9>2015/08/18 21:06:56.05: comp/45 55% of memory in use:  2787115008 bytes (change of 0)",
      "@version" => "1",
    "@timestamp" => "2015-08-18T21:06:56.05Z",
          "host" => "hostname",
          "path" => "/usr/phd/raveesh/sample.log_20150819000609",
          "tags" => [
        [0] "****_grokparsefailure****"
    ]
}

That means your grok expression cannot parse the input.

You should fix the expression according to your input and then the csv will output properly.

Checkout http://grokconstructor.appspot.com/do/match for some help

BTW, are you sure the patterns NSFW, CPU, COMPONENT_ID, ... are defined somewhere ?

HIH