Elasticsearch / Logstash define time or date when importing old log files

Question

I have some old log files (one file per day). log-2017.09.01.json log-2017.09.02.json etc

There is no date information in the json file.

By default, the timestamp of the index is the date of the creation of the index.

I am trying to create an index for each of these log file and I want the timestamp of the index corresponding to each log file to be the same as the one defined by the name of the file. i.e., I want an index "log-2017.09.01" for which the timestamp would be 2017.09.01 and another index "log-2017.09.02" for which the timestamp would be 2017.09.02

Does anyone know how to simply do it ?

Answer 1

There isn't a simple here, but it can be done. It takes a few steps.

The first step, get the date out of the file-path.

filter {
  grok {
    match => { "path", "^log-%{DATA:date_partial}$" }
  }
}

The second step is to pull your timestamp data out of the log-lines. I'm assuming you know how to do that.

The third step is to assemble a date field out of parts.

filter {
  mutate {
    add_field => { "full_timestamp", "%{date_partial} %{date_hour}:%{date_minute}" }
  }
}

The last step is to use the date{} filter on that constructed field.

filter {
  date {
    match => [ "full_timestamp", "yyyy.MM.dd HH:mm" ]
  }
}

This should give you an idea as to the technique needed.