Search code examples
elasticsearchpodman

Podman with Elasticsearch Permissions Issues - Unable to access 'path.data' (/usr/share/elasticsearch/data)

Now that RHEL8 is out Docker is no longer a viable option so I'm trying to get a cluster running in Podman, but have run into a permissions error. For whatever reason, no matter what I try, I get java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)

Directory setup

  mkdir /var/elasticsearch-data
  mkdir /var/elasticsearch-data/data01
  mkdir /var/elasticsearch-data/data02
  mkdir /var/elasticsearch-data/data03
  mkdir /var/elasticsearch-data/data04

  chmod -R g+rwx /var/elasticsearch-data
  chgrp -R 0 /var/elasticsearch-data

I eventually just gave up and set the directory with chmod -R 777 /var/elasticsearch-data and the error persists. I've tried changing the directory ownership to non-privileged users and I've tried changing the group.

Podman Compose File

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03,es04
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02,es04
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data03:/usr/share/elasticsearch/data
    networks:
      - elastic
  es04:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es04
    environment:
      - node.name=es04
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02,es03
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data04:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana:
    image: docker.elastic.co/kibana/kibana:7.7.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200, http://es02:9200, http://es03:9200, http://es04:9200
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  data04:
    driver: local

networks:
  elastic:
    driver: bridge

Run with podman-compose

podman-compose up

Log Output

[root@mapsdemo opt]# podman-compose up
podman pod create --name=opt --share net -p 9200:9200 -p 5601:5601
Error: unable to create pod: error adding pod to state: name opt is in use: pod already exists
125
podman create --name=es01 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es01 -e node.name=es01 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es02,es03 -e cluster.initial_master_nodes=es01,es02,es03 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data01,destinatio
n=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03:127.0.0.1
 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es01" is already in use by "fc3902dad1e80d30d0cb778f2667356a26e2b005a7b24de032aad13ca7147622". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=es02 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es02 -e node.name=es02 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es01,es03,es04 -e cluster.initial_master_nodes=es01,es02,es03,es04 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data02,
destination=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03
:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es02" is already in use by "e66f93a50e61e62a1f0408aee3bef96f8ddf949f7edb35bfdbdf5557da91f743". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=es03 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es04 -e node.name=es04 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es01,es02,es03 -e cluster.initial_master_nodes=es01,es02,es03,es04 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data04,
destination=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03
:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es03" is already in use by "f707acc5bd286dcab77a5463b5d3ce04bceb55705203025926a87180b8ad8c2b". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=kib01 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=kibana -e ELASTICSEARCH_UR
L=http://es01:9200 -e ELASTICSEARCH_HOSTS=http://es01:9200, http://es02:9200, http://es03:9200, http://es04:9200 --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.
0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/kibana/kibana:7.7.0
Error: error creating container storage: the container name "kib01" is already in use by "3cc0578669211247b729d82cb4993c679164ebd5ea32858818b737bd6dcac293". You have to remove that container to be able to reuse that name.: that name is alr
eady in use
125
podman start -a es01
podman start -a es02
podman start -a es03
podman start -a kib01
{"type": "server", "timestamp": "2020-05-25T00:32:32,478Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate
 memory" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "This can result in part of the JVM being swapped out." }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 655
36" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "These can be adjusted by modifying /etc/security/limits.co$
f, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,482Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "If you are logged in interactively, you will have to re-log
in for the new limits to take effect." }
{"type": "server", "timestamp": "2020-05-25T00:32:32,537Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "uncaught exception in thread [m
ain]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)",

Has anyone else run into this?

Solution

  • if you run the following:

    chcon -t svirt_sandbox_file_t <host file path>

    This should enable you to use selinux (which is a good idea)