Logstash Get the value from [@metadata] in logstash filter

Question

i have in s3 this file structure :

mylogs/b222/foo/32/*.log
mylogs/b222/foo/33/*.log
mylogs/b233/foo11/33/*.log

i like to extract the full file path but no matter i do can't get it using @metadata this is my configuration :

input {
      s3 {
        "access_key_id" => "xxxx"
        "secret_access_key" => "xxxx"
        "bucket" => "perso-logstash-logs"
        "additional_settings" => {
        "force_path_style" => true
        "follow_redirects" => false
        }
        
      }
    }
    filter {
    # parse pserver line
     
     
     mutate { add_field => { "file0" => "[@metadata]" } }
     mutate { add_field => { "file1" => "%{[@metadata]}" } }
     mutate { add_field => { "file2" => "%{[@metadata][key]}" } }
    

    
    }
    output {
      stdout { codec => rubydebug }
      elasticsearch {
        index => "%{version}-%{projname}-%{myId}-%{+YYYY.MM.dd}"
        hosts => [ "http://xxxx.svc:9200" ]
        user => "elastic"
        password => "xxxxxx"
        codec => "json"
      }
    }

And here is the output I'm getting in the log : only file1 gives me the name of the log file , but i like to get all the path : for example :

mylogs/b222/foo/32/foo.2021-01-07.0.log

{
    "@timestamp" => 2021-01-18T09:34:24.586Z,
     "@metadata" => {
        "s3" => {
            "key" => "foo.2021-01-07.0"
        }
    },
      "@version" => "1",
         "file2" => "%{[@metadata][key]}",
         "file0" => "[@metadata]",
         "file1" => "{\"s3\":{\"key\":\"foo.2021-01-07.0.log\"}}",
       "message" => "2021-01-07 08:58:55.519 [localhost-startStop-1] INFO  ......"
}

Answer 1

Well it seems like the full path is not part of the event, hence you can not extract it from any field. Which fields get exported is part of the input filter implementation.

See the comments in the following link as an additional reference: Extracting fields from AWS S3 input paths in logstash

If the file path is static you could add it to the filename as a prefix.

Btw:

As you can see from

"file1" => "{"s3":{"key":"foo.2021-01-07.0.log"}}",

the field "key" is nested inside another field called "s3".

So if you want to extract the file name

mutate { add_field => { "file2" => "%{[@metadata][s3][key]}" } }

should do the trick.

I hope I could help you.