Search code examples
elasticsearchelasticsearch-curator

How to delete indexes automatically regularly in ES?

I am using ES 2.3.3 and Logstash 2.3.3. I have been using Logstash to send data and map them into ES for indexing, i.e.logstash-{Date}. And I would only like to keep the file which is the latest 1 year. Any index over a year should be deleted. I was using 3.5.1 before. The way for me to delete the indexes is to input a command everyday.

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
   --timestring '%Y.%m.%d'

Recently, I then I have upgraded curator 3.5.1 to curator 4. However I could not find where curator is stored even though I have read through the examples from https://www.elastic.co/guide/en/elasticsearch/client/curator/current/command-line.html Therefore, I would wanna know where will the configuration file be and why there will be a missing of action_file? Does that mean I need to create a new .curator directory and also my own curator.yml and action.yml file?

And after I have created my action.yml file, should I just follow the https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html#ex_delete_indices and add this part into my action.yml file in order to delete logstash indexes over a year?

Thanks

Solution

  • The configuration file can be anywhere, so long as you launch Curator with the --config flag:

    curator --config /path/to/curator_config.yml

    However, if you make a .curator path in the home directory of the user who will be running Curator (via cron, ostensibly), it will look there for a file called curator.yml, e.g. /home/username/.curator/curator.yml

    With that file properly configured in that location, Curator will not require the --config flag.

    Curator just uses the final argument as the action file:

    » curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

    An example of running Curator with the default configuration file in $HOME/.curator/curator.yml would be:

    curator /path/to/actionfile.yml

    And with a custom configuration file:

    curator --config /path/to/curator_config.yml /path/to/actionfile.yml

    Following the action file examples is a great place to start. Feel free to experiment with new configurations, but be sure to use the --dry-run flag when doing so, to prevent any action from being taken while testing.