What is a correct way to access fields in logstash configuration file?

Question

I have many services (platforms), deployed on kubernetes or VMs. Filebeats ships logs to logstash for each of service. I have to set up different indices for logs came from k8s and vm [log_source], but index must also include the name of the service [log_type].

I have the configuration only with [log_type] that works fine:

logstash.config (output part)

output {
    if ([fields][log_type] == "service-1"){
        elasticsearch {
            hosts => ["https://localhost:9200"]
            manage_template => false
                    user => '${ELASTICSEARCH_USERNAME}'
                    password => '${ELASTICSEARCH_PASSWORD}'
            index => "filebeat-service-1-%{+YYYY.MM.dd}"
        }       
    } else if ([fields][log_type] == "service-2") {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            manage_template => false
                    user => '${ELASTICSEARCH_USERNAME}'
                    password => '${ELASTICSEARCH_PASSWORD}'
            index => "filebeat-service-2-%{+YYYY.MM.dd}"
        }
    } else {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            manage_template => false
                    user => '${ELASTICSEARCH_USERNAME}'
                    password => '${ELASTICSEARCH_PASSWORD}'
            index => "filebeat-default-%{+YYYY.MM.dd}"
        }
    }
}

The part of filebeat.yml file, that allows to send different log_sources:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
    - /var/log/*.log
    include_lines: ['DEBUG', 'WARN', 'INFO', 'ERROR', 'WARNING', 'FATAL', 'CRITICAL']
    exclude_files: ['.gz$']
    multiline.pattern: ^\d
    multiline.negate: true
    multiline.match: after
    fields:
      log_type: service-1

Can I somehow optimize logstash.config file to parametrize it? Because it will be many different services in future. Can I do like this?

output {
    elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        manage_template => false
        user => '${ELASTICSEARCH_USERNAME}'
        password => '${ELASTICSEARCH_PASSWORD}'
        cacert => ["/usr/share/logstash/config/certs/ca.pem"]
        index => "filebeat-[fields][log_type]-%{+YYYY.MM.dd}"
    }       
}
    

Will [fields][log_type] be replaced by what I specify in the filebeat config (e.g. service-1)?

Answer 1

Yes, you can set dynamic index name by using value from field like that

output {
    elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        manage_template => false
        user => '${ELASTICSEARCH_USERNAME}'
        password => '${ELASTICSEARCH_PASSWORD}'
        cacert => ["/usr/share/logstash/config/certs/ca.pem"]
        index => "filebeat-%{[fields][log_type]}-%{+YYYY.MM.dd}"
    }       
}

According to the documentation: plugins-outputs-elasticsearch-index

The index to write events to. This can be dynamic using the %{foo} syntax

And from this document: Writing to different indices: best practices