Search code examples
loggingelasticsearchbackfillelastic-stack

ELK stack - How do i backfill all old logs into elasticsearch?

Ive got ELK + filebeat set up with logs coming in, but want to backfill all the old logs into logstash/ES. How can I do this? People have mentioned deleting sincedb file and/or adding this to my logstash input.conf:

  file {
      path => "/var/log/xx/xx.log"
      start_position => "beginning"
      sincedb_path => "/dev/null"
     }

But I added it to my input conf and restarted logstash and still don't see old logs in Kibana. I also can't find the sincedb_* file that everyone refers to. My ELK node is a RHEL server.

Thanks

Solution

  • If those files are old, it's a good idea to also add ignore_older => 0 to your config.

      file {
      path => "/var/log/ptsfd-mms/ptsfd-mms.log"
      start_position => "beginning"
      sincedb_path => "/dev/null"
      ignore_older => 0
  }