Search code examples
elasticsearchfilebeat

How to specify pipeline for Filebeat Nginx module?


I have web server (Ubuntu) with Nginx + PHP.
It has Filebeat, which sends Nginx logs to Elastic ingestion node directly (no Logstash or anything else).
When I just installed it 1st time, I made some customizations to the pipeline, which Filebeat created. Everything worked great for a month or so.

But I noticed, that every Filebeat upgrade result in the creation of new pipeline. Currently I have these:

filebeat-7.3.1-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-error-pipeline: {},
filebeat-7.2.0-nginx-access-default: {},
filebeat-7.3.2-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-access-default: {},
filebeat-7.3.1-nginx-access-default: {},
filebeat-7.3.2-nginx-access-default: {},
filebeat-7.2.0-nginx-error-pipeline: {}

I can create new pipeline, but how do I tell (how to configure) Filebeat to use specific pipeline?

Here is what I tried and it doesn't work:

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/*/*access.log"]

    # Convert the timestamp to UTC
    var.convert_timezone: true

    # The Ingest Node pipeline ID associated with this input. If this is set, it
    # overwrites the pipeline option from the Elasticsearch output.
    output.elasticsearch.pipeline: 'filebeat-nginx-access-default'
    pipeline: 'filebeat-nginx-access-default

It still using filebeat-7.4.1-nginx-error-pipeline pipeline.

Here is Filebeat instructions on how to configure it (but I can't make it work): https://github.com/elastic/beats/blob/7.4/filebeat/filebeat.reference.yml#L1129-L1130

Question: how can I configure Filebeat module to use specific pipeline?

Update (Nov 2019): I submitted related bug: https://github.com/elastic/beats/issues/14348


Solution

  • In beats source code, I found that the pipeline ID is settled by the following params:

    • beats version
    • module name
    • module's fileset name
    • pipeline filename

    the source code snippet is as following:

    // formatPipelineID generates the ID to be used for the pipeline ID in Elasticsearch
    func formatPipelineID(module, fileset, path, beatVersion string) string {
        return fmt.Sprintf("filebeat-%s-%s-%s-%s", beatVersion, module, fileset, removeExt(filepath.Base(path)))
    }
    

    So you cannot assign the pipeline ID, which needs the support of elastic officially.

    For now, the pipeline ID is changed along with the four params. You MUST change the pipeline ID in elasticsearch when you upgrading beats.