IIS Logs and Event Logs

Question

First off thank you for any advice and your time.

I recently setup an Elk stack for the company I just started working for. (This is my first experience using Logstash and Nxlog.) What I would like to do is send both IIS logs and EventLogs from the same webserver to logstash using nxlog.

I just don't understand how to send two types of logs from one source and have the logstash.conf filter this data correctly.

This is my nxlog.conf

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module xm_json
</Extension>
<Input iis_1>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>  
<Input iis_2>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC2\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>
<Input iis_4>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC4\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>  
<Input eventlog>
        Module im_msvistalog
        Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Output out_iis>  
    Module  om_tcp
    Host    10.191.132.86
    Port    5555
    OutputType  LineBased
</Output>  
<Route 1>  
    Path    iis_1, iis_2, iis_4, eventlog=> out_iis
</Route> 

My Current logstash.conf

input {  
      tcp {
              type => "iis"
              port => 5555
              host => "10.191.132.86"
      }
}
filter {  
    if [type] == "iis" {
        grok {
            match => ["@message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
        }
    }
}
output {  
    elasticsearch {
    protocol => "http"
        host => "10.191.132.86"
        port => "9200"
    }
}

It looks like you can filter different data by setting the type and doing if type else this type. But if they are coming from the same source how do I specify different types?

:) Thanks!

Answer 1

A way to do this is filter by a known record entry in each log and wont exist in the other, for example [cs_bytes etc]:

e.g.

if [iisfield] {
   mark type as IIS
else 
   mark type as EventLog
}

I have written a IIS and Event log agent that captures logs for Logit.io they might already do everything you already want