Search code examples
elasticsearchelasticsearch-5elasticsearch-7

Elasticsearch 7.10 : How to safely remove a node and free up space?


As I am running out of my disk space, I want to remove all the indices from the node. Note that I have just one node, and I am running the elastic server on my local machine.

Basically, when I start the elastic server, it fails with the following log -

[2020-12-13T22:06:51,839][WARN ][o.e.c.r.a.DiskThresholdMonitor] [sedna] flood stage disk watermark [95%] exceeded on [k7t835EJQ6KzPu-7a-aW8Q][sedna][/home/ssahoo/elasticsearch-7.10.0/data/nodes/0] free: 879mb[0.8%], all indices on this node will be marked read-only

While on the terminal I get the following error -

Could not rename log file 'logs/gc.log' to 'logs/gc.log.05' (Reserved).
(base) ssahoo@sedna:~/elasticsearch-7.10.0$ [2020-12-14T17:07:17,831][WARN ][stderr                   ] [sedna] Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Input/output error
2020-12-14 17:07:17,848 main ERROR Unable to move file /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch_server.json to /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.json: java.nio.file.FileSystemException /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch_server.json -> /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.json: Disk quota exceeded
2020-12-14 17:07:17,852 main ERROR Unable to copy file /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch_server.json to /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.json: java.nio.file.FileSystemException /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.json: Disk quota exceeded
2020-12-14 17:07:17,857 main ERROR Unable to move file /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch.log to /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.log: java.nio.file.FileSystemException /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch.log -> /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.log: Disk quota exceeded
2020-12-14 17:07:17,861 main ERROR Unable to copy file /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch.log to /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.log: java.nio.file.FileSystemException /home/ssahoo/elasticsearch-7.10.0/logs/elasticsearch-2020-12-13-1.log: Disk quota exceeded
[2020-12-14T17:07:17,863][WARN ][stderr                   ] [sedna] Likely root cause: java.io.IOException: Input/output error
[2020-12-14T17:07:17,864][WARN ][stderr                   ] [sedna]     at java.base/java.io.FileDescriptor.close0(Native Method)
[2020-12-14T17:07:17,864][WARN ][stderr                   ] [sedna]     at java.base/java.io.FileDescriptor.close(FileDescriptor.java:297)

Therefore I want to free up space by removing all the indices, and therefore the node. So how can I remove the single node that I have?

Or if may I ask what would your suggestion be to overcome this? I can't afford additional disk space and is my major constraint.

Thank you.


Solution

  • I want to remove all the indices from the node

    In order to remove all indices from a node, get the list of indices and then call delete on each.

    E.g.

    for i in $(curl -s -u user:password -XGET "<your_es_host_name>:9200/_cat/indices?s=index&h=index"); do echo "deleting Index --> $i"; curl -s -u user:password -XDELETE "< your_es_host_name>:9200/$i"; done
    

    This will delete all your indices. If you want to exclude certain indices, you can modify the code to exclude indices like "<your_es_host_name>:9200/_cat/indices?s=index&h=index | grep -v exclude_index".

    Hope this helps.