We've moved from AWS Elasticsearch 2.3 to 5.1 and found that our old curator 3 command no longer worked in deleting indices older than 30 days. Our old command which worked looks as follows:
00 00 * * 1 /bin/curator --host elasticsearch.production.domain.aws --port 80 delete indices --older-than 30 --time-unit days --timestring %Y.%m.%d --exclude .kibana
Moving to Curator 4.2.6 was required to support ES 5.1 however, when using the example provided from the elasticsearch page, we found we received the error:
2017-02-15 11:46:18,874 INFO Preparing Action ID: 1, "delete_indices"
2017-02-15 11:46:18,884 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-02-15 11:46:18,897 ERROR Failed to complete action: delete_indices. <class 'KeyError'>: 'settings'
Anyone else had success with that configuration after migrating?
I've defined separately the curator.yaml configuration which I believe is correct:
client:
hosts:
- elasticsearch.production.domain.aws
port: 80
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
TL;DR: AWS ES still does not support the necessary API call (well, they do, but it doesn't return the necessary data) for Curator to work with it.
Though it does seem that AWS ES has added the necessary /_cluster/state endpoint (which is why Curator 4 did not support AWS ES 2.3), they seem to have omitted some of the necessary data from that endpoint (that's the part where you get <class 'KeyError'>: 'settings'. The settings sub-object in the JSON response is missing, so Curator cannot complete its task.
There's already an open issue in Curator for this: https://github.com/elastic/curator/issues/880, though I do not think there is anything that Curator can do to overcome this.