Can't create a field with a variable from a grok match regex

Question

I am currently using logstash, elasticsearch and kibana 6.3.0

My log are generated at a unique id path : /tmp/USER_DATA/FactoryContainer/images/(my unique id)/oar/oar_image_job(my unique id).stdout

What I want to do is to match this unique id and to create a field with this id.

I m a bit novice to logstash filter but I don't know why it doesn't want to use my uid and always return me %{uid} in my field or this Failed to execute action error.

my filter :

input {
  file {
    path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
    start_position => "beginning"
    add_field => { "data_source" => "oar-image-job" }
   }
}

filter {
    grok {
        match => ["path","%{UNIXPATH}%{NUMBER:uid}%{UNIXPATH}"]
    }
    mutate {
        add_field => [ "unique_id" => "%{uid}" ]
    }
}

output {
  if [data_source] == "oar-image-job" {
    elasticsearch {
        index => "oar-image-job-%{+YYYY.MM.dd}"
        hosts => ["localhost:9200"]
        }
    }
}

the data_source field is to avoid this issue: When you put multiple config files in a directory for Logstash to use, they will all be concatenated

in the grok debugger %{UNIXPATH}%{NUMBER:uid}%{UNIXPATH} my path return me the good value

Answer 1

link to the solution : https://discuss.elastic.co/t/cant-create-a-field-with-a-variable-from-a-grok-match-regex/142613/7?u=thesmartmonkey

the correct filter :

input {
  file {
    path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
    start_position => "beginning"
    add_field => { "data_source" => "oar-image-job" }
   }
}

filter {
    grok {
        match => { "path" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:unique_id}/oar/oar_image_job%{DATA}.stdout" ] }
    }

}

output {
  if [data_source] == "oar-image-job" {
    elasticsearch {
        index => "oar-image-job-%{+YYYY.MM.dd}"
    hosts => ["localhost:9200"]
    }
    }
}