Filebeat Input Fields are not sent to Logstash

Question

StackOverflow community!

I am trying to collect some system logs using Filebeat and then further process them with LogStash before viewing them in Kibana.

Now, as I have different logs location, I am trying to add specific identification fields for each of them within filebeat.yml.

- type: log
   enabled: true
   paths:
     - C:\Users\theod\Desktop\Logs\Test2\*
processors:
 - add_fields:
    target: ''
   fields: 
    name:"drs"


 - type: log
   enabled: true
   paths:
     - C:\Users\theod\Desktop\Logs\Test\*
processors:
 - add_fields:
    target: ''
   fields: 
    name:"pos"

Depending on that, I am trying to apply some Grok filters in the Logstash conf file:

input {
  beats {
    port => 5044
  }
}


filter
{    
if "pos" in [fields][name] {
        grok {
            match => { "message" => "\[%{LOGLEVEL:LogLevel}(?: ?)] %{TIMESTAMP_ISO8601:TimeStamp} \[%{GREEDYDATA:IP_Address}] \[%{GREEDYDATA:Username}] %{GREEDYDATA:Operation}] \[%{GREEDYDATA:API_RequestLink}] \[%{GREEDYDATA:Account_name_and_Code}] \[%{GREEDYDATA:Additional_Info1}] \[%{GREEDYDATA:Additional_Info2}] \[%{GREEDYDATA:Store}] \[%{GREEDYDATA:Additional_Info3}](?: ?)%{GREEDYDATA:Error}" }
        }   
}
if "drs" in [fields][name] {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601:TimeStamp} \[%{DATA:Thread}] %{LOGLEVEL:LogLevel} (?: ?)%{INT:Sequence} %{DATA:Request_Header}] %{GREEDYDATA:Request}" }
        }   
}   
}



output
{
if "pos" in [fields][name] {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "[fields][name]logs-%{+YYYY.MM.dd}"
    }
}
else if "pos" in [fields][name] {
elasticsearch {
    hosts => ["localhost:9200"]
    index => "[fields][name]logs-%{+YYYY.MM.dd}"
    }
} else {
elasticsearch {
    hosts => ["localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    }
}
}

Now, every time I'm running this, the conditionals in the Logstash conf are ignored. Checking the Filebeat logs, I'm noticing that no fields are sent to Logstash.

Can someone offer some guidance and perhaps point out what am I doing wrong?

Thank you!

Answer 1

Your Filebeat config is not adding the field [fields][name], it is adding the field [name] in the top-level of your document because of your target configuration.

processors:
 - add_fields:
   target: ''
   fields: 
    name:"pos"

All your conditional test the field [fields][name], which does not exist.

Change your conditionals to test the [name] field.

if "pos" in [name] { 
    ... your filters ...
}