Search code examples
logstashlogstash-forwarder

Possible to specify two different codecs in lumberjack?

I have just put up an ELK stack, but I am having trouble regarding the logstash configuration in /etc/logstash/conf.d I have two input sources being forwarded from one linux server, which has a logstash forwarder installed on it with the "files" looking like:

{
      "paths": ["/var/log/syslog","/var/log/auth.log"],
      "fields": { "type": "syslog" }
    },
    { 
      "paths": ["/var/log/osquery/osqueryd.results.log"],
      "fields": { "type": "osquery_json" } 
}

As you can see, one input is an osquery output (json formatted), and the other is syslog. My current config for logstash is osquery.conf:

input {
  lumberjack {
    port => 5003
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    codec => "json"
  }
}

filter {
   if [type] == "osquery_json" {
      date {
        match => [ "unixTime", "UNIX" ]
      }
   }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Which works fine for the one input source, but I do not know how to add my other syslog input source to the same config, as the "codec" field is in the input -- I can't change it to syslog...

I am also planning on adding another input source in a windows log format that is not being forwarded by a logstash forwarder. Is there anyway to structure this differently?

Solution

  • It's probably better to just remove the codec from your input if you are going to be handling different codecs on the same input:

    input {
  lumberjack {
    port => 5003
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
   if [type] == "osquery_json" {
      json {
        source => "field_name_the_json_encoded_data_is_stored_in"
      }
      date {
        match => [ "unixTime", "UNIX" ]
      }
   }
   if [type] == "syslog" {

   }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

    Then you just need to decide what you want to do with your syslog messages.

    I would suggest also splitting your config into multiple files. I tend to to use 01-filename.conf - 10-filename.conf for inputs, 11-29 as filters and anything above that for outputs. These files will be loaded in to logstash in the order they are printed in an ls.