How to configure Elasticsearch Index Lifecycle Management (ILM) in eck-operator

Question

I use eck-operator (with eck-operator-crd) from official elastic repository https://artifacthub.io/packages/helm/elastic/eck-operator Kibana and Elasticsearch manifests are deploying and working fine. My question is how to configure additional features like: Index Lifecycle Policy, Kibana Data View, New Templates, Data Streams, Dashboards etc.? I saw few operators with kind of manifests but nothing from mentioned above. Thanks for any remarks.

Answer 1

We do that via config maps. We store the index templates and index life cycle policies as json format. Also an execution script is a part of config map which pushes the index templates and ilm policies to elasticsearch via POST. Since everything is inside the volume now, we then create a cron job to execute this script.

For example this is the index template all-indices.json:

{
    "index_patterns": ["abc*"],
    "priority": 300,
    "composed_of": ["template1", "template2"], 
    "version": 3
}

execution-script.sh

 curl -XPUT "http://some_url:9200/_index_template/all-indices" -H 'Content-Type: application/json' -d @/etc/elasticsearch-templates/all-indices.json  

We create these config map from terraform.

kibana.yaml

...
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
        - name: elasticsearch-templates
          mountPath: /etc/elasticsearch-templates
          readOnly: true
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "sleep 15 && sh /etc/elasticsearch-templates/execution-script.sh"]
      volumes:
        - name: elasticsearch-templates
          configMap:
            name: ilm-and-index-templates

Then the cron job executes the script

apiVersion: batch/v1
kind: CronJob
metadata:
  name: script-execution
spec:
  schedule: "0 5 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: script-execution
            image: alpine/curl:latest
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - sh /etc/elasticsearch-templates/execution-script.sh
            volumeMounts:
            - name: elasticsearch-templates
              mountPath: /etc/elasticsearch-templates
              readOnly: true
          restartPolicy: OnFailure
          volumes:
            - name: elasticsearch-templates
              configMap:
                name: ilm-and-index-templates