Search code examples
logstashlogstash-configuration

Add log4net Level field to logstash.conf file

I'm trying to add LEVEL field (so it shows up in Kibana). My logstash.conf

Input:

2018-03-18 15:43:40.7914 - INFO: Tick 
2018-03-18 15:43:40.7914 - ERROR: Tock

file:

input {
  beats {
    port => 5044
  }
}
filter {
  grok {      
      match => { 
            "message" => "(?m)^%{TIMESTAMP_ISO8601:timestamp}~~\[%{DATA:thread}\]~~\[%{DATA:user}\]~~\[%{DATA:requestId}\]~~\[%{DATA:userHost}\]~~\[%{DATA:requestUrl}\]~~%{DATA:level}~~%{DATA:logger}~~%{DATA:logmessage}~~%{DATA:exception}\|\|"
        }
      match => {
        "levell" => "(?m)^%{DATA:level}"
      }
      add_field => { 
        "received_at" => "%{@timestamp}" 
        "received_from" => "%{host}"
        "level" => "levell"
      }
      remove_field => ["message"]      
    }
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss:SSS" ]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    sniffing => true
    index => "filebeat-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

this prints out "levell" instead of "INFO/ERROR" etc

EDIT: Input:

2018-03-18 15:43:40.7914 - INFO: Tick

configuration:

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}
filter {
  grok {      
      match => { "message" => "(?m)^%{TIMESTAMP_ISO8601:timestamp}~~\[%{DATA:thread}\]~~\[%{DATA:user}\]~~\[%{DATA:requestId}\]~~\[%{DATA:userHost}\]~~\[%{DATA:requestUrl}\]~~%{DATA:level}~~%{DATA:logger}~~%{DATA:logmessage}~~%{DATA:exception}\|\|" }
      add_field => { 
        "received_at" => "%{@timestamp}" 
        "received_from" => "%{host}"
      } 
    }
  grok {      
      match => { "message" => "- %{LOGLEVEL:level}" }
      remove_field => ["message"]      
    }
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss:SSS" ]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    sniffing => true
    index => "filebeat-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

Output I'm getting. Still missing received_at and level: enter image description here

Solution

  • In that part of the configuration:

      add_field => { 
    "received_at" => "%{@timestamp}" 
    "received_from" => "%{host}"
    "level" => "levell"
  }

    When using "level" => "levell", you just put the String levell in the field level. To put the value of the field named levell, you have to use %{levell}. So in you case, it would look like:

      add_field => { 
    "received_at" => "%{@timestamp}" 
    "received_from" => "%{host}"
    "level" => "%{levell}"
  }

    Also the grok#match, according to the documentation:

    A hash that defines the mapping of where to look, and with which patterns.

    So trying to match on the levell field won't work, since it look like it doesn't exist yet. And the grok pattern you're using to match the message field don't match the example you provided.