Search code examples
elasticsearchlogstashkibanafilebeat

How to have different index name for different log from same filebeat to logstash

I have setup the version(7.3.1) of ELK. filebeat(7.3.1) on a different VM. I have multiple logs on the VM with Filebeat installed on it. I want to have different index name for different logs. I have tried a way which is not working,the config files are as below

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG1/forecaster.log
  fields:
  log_type: type1
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG2/forecaster.log
  fields:
  log_type: type2
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG3/forecaster.log
  fields:
  log_type: type3

logstash.conf

input {
    beats {
                type => "filebeat"
                port => "5044"
    }
}

filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
}
output {
stdout {
    codec => rubydebug
  }
if ([fields][log_type] == "type1") {
elasticsearch {
hosts => ["IP:9200"]
index => "log1"
}
}
if ([fields][log_type] == "type2") {
elasticsearch {
hosts => ["IP:9200"]
index => "log2"
}
}
if ([fields][log_type] == "type3") {
elasticsearch {
hosts => ["IP:9200"]
index => "log3"
}
}
}

using the above configurations and after analyzing the logs of EL and filebeat, log files are fetched from filebeat and send to logstash where it is being processed but they are not sent to elastic search.

I need help figuring out what's wrong/missing in order to make this work

Thanks

Solution

  • It seems that the indenting in the filebeat configuration is not correct in the fields section, you're missing two space characters

    filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG1/forecaster.log
  fields:
    log_type: type1                            <-- fis this line
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG2/forecaster.log
  fields:
    log_type: type2                            <-- fis this line
- type: log
  enabled: true
  paths:
    - /data01/-----/LOG3/forecaster.log
  fields:
    log_type: type3                            <-- fis this line