how to properly configure a different event.dataset per log for filebeat to logstash

Question

I set up a server with kibana 7.11.1 with logstash. I'm new to Kibana so.. I apologize for any mistakes in advance :)

I have another server that has filebeat configured that sends data to logstash.

I can see that data properly in kibana but event.dataset is empty. i would like to set it myself.

by reading Logstash and filebeat set event.dataset value I noticed that I can set it in logstash configuration. but that will add the same value for the all the logs that are going through logstash. but i want to add different values for different type of log files.

I tried setting event.dataset under fields: in filebeat.yml but it did not appear on kibana. i guess these are custom variables and not the proper way to set event.dataset.

in general i have services that are running under pm2, so i want to set event.dataset to include pm2.<LOG_NAME>.<LOG_TYPE>

this is my inputs configuration in filebeat.yml:

- type: log
  enabled: true
  paths:
    - /home/ubuntu/.pm2/*-error-*log
  fields:
   level: error
- type: log
  enabled: true
  paths:
    - /home/ubuntu/.pm2/logs/cdr-ftp-out*log

and this is my logstash configuration:

input {
    beats {
        port => 5544
    }
}
filter {
 grok {
   match => {"message"=>"%{DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\]: %{GREEDYDATA:msg}"} 
 }
mutate {
    rename => ["host", "server"]
    convert => {"server" => "string"}
}
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
    }
}

how to properly resolve this issue?

thanks

Answer 1

The right way to do it is like this:

- type: log
  enabled: true
  paths:
    - /home/ubuntu/.pm2/*-error-*log
  fields:
    level: error
    event.dataset: dataset1                   <--- add this
  fields_under_root: true                     <--- add this
- type: log
  enabled: true
  fields:
    event.dataset: dataset2                   <--- add this
  fields_under_root: true                     <--- add this
  paths:
    - /home/ubuntu/.pm2/logs/cdr-ftp-out*log