Logstash: how to use filter to match filename when using s3

Question

I am new to logstash. I have some logs stored in AWS S3 and I am able to import them to logstash. My question is: is it possible to use the grok filter to add tags based on the filenames? I try to use:

grok { 
    match => {"path" => "%{GREEDYDATA}/%{GREEDYDATA:bitcoin}.err.log"}
    add_tag => ["bitcoin_err"]
    }

This is not working. I guess the reason is "path" only working with file inputs.

Here is the structure of my S3 buckets:

my_buckets
    ----A
        ----2014-07-02
            ----a.log
            ----b.log
    ----B
        ----2014-07-02
            ----a.log
            ----b.log

I am using this inputs conf:

s3 {
    bucket => "my_buckets"
    region => "us-west-1"
    credentials => ["XXXXXX","XXXXXXX"]
    }

What I want is that, for any log messages in:

• "A/2014-07-02/a.log": they will have tag ["A","a"].

• "A/2014-07-02/b.log": they will have tag ["A","b"].

• "B/2014-07-02/a.log": they will have tag ["B","a"].

• "B/2014-07-02/b.log": they will have tag ["B","b"].

Sorry about my english....

Answer 1

There is no "path" in S3 inputs. I mount the S3 storage on my server and use the file inputs. With file inputs, I can use the filter to match the path now.