We have on our elasticsearch several indexes. They come from FluentD pluging sendings logs fron our docker containers. We would like to delete old indexes not only older than specific amount of days based on index name but applying different delete rules depending on log fields.
Here is an example of log:
{
"_index": "fluentd-2018.03.28",
"_type": "fluentd",
"_id": "o98123bcbd_kqpowkd",
"_version": 1,
"_score": null,
"_source": {
"container_id": "bbd72ec5e46921ab8896a05684a7672ef113a79e842285d932f",
"container_name": "/redis-10981239d5",
"source": "stdout",
"log": "34:M 28 Mar 15:07:51.086 * 10 changes in 300 seconds. Saving...\r34:M 28 Mar 15:07:51.188 * Background saving terminated with success\r",
"@timestamp": "2018-03-28T15:07:56.217739954+00:00",
"@log_name": "docker.redis"
},
"fields": {
"@timestamp": [
"2018-03-28T15:07:56.217Z"
]
}
}
In that case, we would like to delete all logs matching @log_name = docker.redis older than 7 days.
Is it possible to define a Curator action which deletes indexes filtered by such a field value?
We tried different filtering without any success. The only action we manage to perform successfully is based on index name:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 30 days
options:
ignore_empty_list: True
disable_action: True
filters:
- filtertype: pattern
kind: prefix
value: fluentd-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
Curator offer only an index level retention configuration. If you need a retention based on document level, you can try with a script that execute a delete by query.
Otherwise, using curator, you need to separate your data in different indexes for applying different retention.