Issue in reading log file that contains date in it's name

Question

I have 2 linux boxes setup in which 1 box contains one component which generates log and logstash installed in it to transfer the logs. And in other box I have redis elasticsearch and logstash. here logstash will act as logstash indexer to grok the data.

Now my problem is that in 1st box component generate new log file everyday, but only difference in log file name varies as per date.

like

counters-20151120-0.log

counters-20151121-0.log

counters-20151122-0.log

and so on, I have included below type of code in my logstash shipper conf file:

file {
    path => "/opt/data/logs/counters-%{YEAR}%{MONTHNUM}%{MONTHDAY}*.log"
    type => "rg_counters"
  }

And in my logstash indexer, I have below type of code to catch those log files:

if [type] == "rg_counters" {
                grok{
                        match => ["message", "%{YEAR}%{MONTHNUM}%{MONTHDAY}\s*%{HOUR}:%{MINUTE}:%{SECOND}\s*(?<counters_raw_data>[0-9\-A-Z]*)\s*(?<counters_operation_type>[\-A-Z]*)\s*%{GREEDYDATA:counters_extradata}"]
        }
    }

output {
elasticsearch { host => ["elastichost1","elastichost1"  ] port => "9200" protocol => "http" }
stdout { codec => rubydebug }
}

Please note that this is working setup and other types log files are getting transfered and processed successfully, so there is no issue of setup.

The problem is how do I process this log file which contains date in it's file name.

Any help here?

Thanks in advance!!

Answer 1

Based on the comments...

Instead of trying to use regexp patterns in your path:

path => "/opt/data/logs/counters-%{YEAR}%{MONTHNUM}%{MONTHDAY}*.log"

just use glob patterns:

path => "/opt/data/logs/counters-*.log"

logstash will remember which files (inodes) that it's seen before.