Search code examples
elasticsearchbitnami

Elasticsearch: Failed to obtain node locks after Installing the bitnami/elasticsearch chart

I use the nfs-csi driver to do persistent storage for bitnami/kafka successfully. But this time I used nfs-csi driver to do persistence for bitnami/elasticsearch but failed.

$ kubectl  get sc
NAME                PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi (default)   nfs.csi.k8s.io    Delete          Immediate           false                  43d

Installing the Chart

$ kubectl create ns elk2

$ helm pull  oci://registry-1.docker.io/bitnamicharts/elasticsearch


$ vim elasticsearch/values.yaml
global:
  imageRegistry: ""
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  ### use the storageclass: nfs-csi
  storageClass: "nfs-csi"   
  elasticsearch:
    service:
      name: elasticsearch
      ports:
        restAPI: 9200
  kibanaEnabled: false

$ helm install elasticsearch ./ -f values.yaml -n elk2 


$ helm list -n elk2
NAME            NAMESPAC    REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
elasticsearch   elk2        1           2023-07-28 22:26:03.468994841 +0800 CST deployed    elasticsearch-19.10.6   8.9.0

‘’

$ kubectl logs -f elasticsearch-master-1 -n elk2
.....
.....
3-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [vector-tile]
[2023-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-expression]
[2023-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-eql]
[2023-07-28T14:29:10,660][ERROR][o.e.b.Elasticsearch      ] [elasticsearch-master-1] fatal exception while booting Elasticsearch java.lang.IllegalStateException: failed to obtain node locks, tried [/bitnami/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)
Caused by: java.io.IOException: failed to obtain lock on /bitnami/elasticsearch/data
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/node.lock
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:825)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)
    ... 7 more
    Suppressed: java.nio.file.AccessDeniedException: /bitnami/elasticsearch/data/node.lock
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
        at java.base/java.nio.file.Files.createFile(Files.java:658)
        at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)
        ... 10 more

ERROR: Elasticsearch did not exit normally - check the logs at /opt/bitnami/elasticsearch/logs/elastic.log

ERROR: Elasticsearch exited unexpectedly

I see the error in the log. But I don't know how to fix it. The authority of nfs is no problem.

Sorry, I'm getting desperate. I have spent days on this problem. I submitted an issue to github, but haven't gotten a response yet.

Solution

  • I got the same error.

    I solved it by adding volumePermissions.enabled: "true" as indicated by this comment https://github.com/bitnami/charts/issues/17834#issuecomment-1776865043 .