Logstash: Dynamic field names based on filename

Question

I got a filename in the format <key>:<value>-<key>:<value>.log like e.g. pr:64-author:mxinden-platform:aws.log containing logs of a test run.

I want to stream each line of the file to elasticsearch via logstash. Each line should be treated as a separate document. Each document should get the fields according to the filename. So e.g. for the above example let's say log-line 17-12-07 foo something happened bar would get the fields: pr with value 64, author with value mxinden and platform with value aws.

At the point in time, where I write the logstash configuration I do not know the names of the fields.

How do I dynamically add fields to each line based on the fields contained in the filename?

The static approach so far is:

filter {
  mutate { add_field => { "file" => "%{[@metadata][s3][key]}"} }
  else {
    grok { match => { "file" => "pr:%{NUMBER:pr}-" } }
    grok { match => { "file" => "author:%{USERNAME:author}-" } } 
    grok { match => { "file" => "platform:%{USERNAME:platform}-" } }
  }
}

Changes to the filename structure are fine.

Answer 1

Answering my own question based on @dan-griffiths comment:

Solution for a file like pr=64,author=mxinden,platform=aws.log is to use the Elasticsearch kv filter like e.g.:

  filter {
    kv {
      source => "file"
      field_split => ","
    }
  }

where file is a field extracted from the filename via the AWS S3 input plugin.