Can I delete a file after it is read, or notify when all are processed?

Question

I have a list of files being read into Elasticsearch through Logstash. I wanted to know how to know when the files were all caught up.

I was thinking of deleting the file after it is fully read in.

I havent seen anything though in regards to notification or ack of file completion, or file deletion. Id love some insight, as i figure it would be a part of the config file. I just have no insight.

Ideally: I would love delete the file after it is fully consumed. That way, i can work my way through all of the file types, starting with txt files.

Answer 1

Using the file input in logstash you can do that, you need to change two config options, mode and file_completed_action.

You need to change the mode option to read, the default is tail, and add the file_completed_action with the value of delete.

file {
    mode => "read"
    path => "/path/to/your/files/*.log"
    file_completed_action => "delete"
}