Kibana can't connect to Elasticsearch using Letsencrypt signed certs
I am trying to run a 3 node elasticsearch cluster with kibana using letsencrypt certs. I have copy pasted the standard docker-compose.yml and environment file from the official elasticsearch documentation here, and successfully got it working with self signed certs without error. When i try and swap out the self signed certs for lets encrypt signed certs the elasticsearch cluster works but kibana stops working with the error
kibana_1 | [2022-09-12T18:52:55.669+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate
The letsencrypt signed certs are all properly mounted into the container with the same owner/group and permissions as the self signed certs, for example
-rw-r----- 1 root root 991 Sep 12 14:03 bundle.zip
drwxr-x--- 2 root root 4096 Sep 12 13:38 ca
-rw-r----- 1 root root 2512 Sep 12 13:38 ca.zip
-rw-r----- 1 root root 1899 Sep 12 14:03 cert.pem
-rw-r----- 1 root root 7610 Sep 12 13:38 certs.zip
-rw-r----- 1 root root 3749 Sep 12 14:03 chain.pem
drwxr-x--- 2 root root 4096 Sep 12 13:38 es01
drwxr-x--- 2 root root 4096 Sep 12 13:38 es02
drwxr-x--- 2 root root 4096 Sep 12 13:38 es03
-rw-r----- 1 root root 5648 Sep 12 14:03 fullchain.pem
-rw-r----- 1 root root 272 Sep 12 13:38 instances.yml
-rw-r----- 1 root root 1826 Sep 12 14:03 intermediary.pem
-rw-r----- 1 root root 1704 Sep 12 14:03 privkey.pem
-rw-r----- 1 root root 1923 Sep 12 14:03 root.pem
Which i achieved by running these commands as suggested in the official docs
sudo find certs/ -type f -exec chmod 640 "{}" \;
sudo find certs -type d -exec chmod 750 "{}" \;
The intermediary.pem and root.pem certs were split out from the fullchain.pem cert and tried as part of the CA cert bundle as suggested in this other SO question. I have tried many different combinations of these certs in both the elastic search and kibana config and although more than one way works for the elastic search nodes, none of them work with kibana.
This is the final attempt i made, using privkey.pem as the key, fullchain.pem as the cert, and chain.pem as the CA, as suggested here in the elasticsearch docs. The below file omits the "setup" container that you will see in the official docs.
version: "2.2"
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata01:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es02,es03
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
es02:
depends_on:
- es01
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata02:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
environment:
- node.name=es02
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es01,es03
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
es03:
depends_on:
- es02
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata03:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
environment:
- node.name=es03
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es01,es02
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
es01:
condition: service_healthy
es02:
condition: service_healthy
es03:
condition: service_healthy
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
volumes:
- kibanadata:/usr/share/kibana/data
- ./certs:/usr/share/kibana/config/certs
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVER_HOST=0.0.0.0
- SERVERNAME=dev.mysite.com
- ELASTICSEARCH_HOSTS=https://dev.mysite.com:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/chain.pem
${MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://dev.mysite.com:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
volumes:
esdata01:
driver: local
esdata02:
driver: local
esdata03:
driver: local
kibanadata:
driver: local
# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=asdf1234
# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=asdf1234
# Version of Elastic products
STACK_VERSION=8.4.1
# Set the cluster name
CLUSTER_NAME=docker-cluster
# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial
# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200
# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80
# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824
# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject
I have verified that elasticsearch is working both by the lack of errors in the output of the containers and also by running
curl -u elastic:asdf1234 https://dev.mysite.com:9200/_cluster/health
which gives me the output
{"cluster_name":"docker-cluster","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":11,"active_shards":22,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
Unlike Elasticsearch for some reason in Kibana you cannot use any of the certs that letsencrypt gives you as the CA. You must find the public root CA cert of letsencrypt itself, which is isrgrootx1.pem and can be downloaded from letsencrypt.org/certs/isrgrootx1.pem. Unfortunately none of this is clear in any documentation anywhere and after days of fruitless searching I stumbled upon this in another SO question!
Once you have that cert you can bind it into the container and then update the config to look like this
version: "2.2"
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata01:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es02,es03
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
es02:
depends_on:
- es01
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata02:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
environment:
- node.name=es02
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es01,es03
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
es03:
depends_on:
- es02
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata03:/usr/share/elasticsearch/data
- ./certs:/usr/share/elasticsearch/config/certs
environment:
- node.name=es03
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es01,es02
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/privkey.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/fullchain.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/chain.pem
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert /usr/share/elasticsearch/config/certs/chain.pem https://dev.mysite.com:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
es01:
condition: service_healthy
es02:
condition: service_healthy
es03:
condition: service_healthy
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
volumes:
- kibanadata:/usr/share/kibana/data
- ./certs:/usr/share/kibana/config/certs
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVER_HOST=0.0.0.0
- SERVERNAME=dev.mysite.com
- ELASTICSEARCH_HOSTS=https://dev.mysite.com:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/isrgrootx1.pem
- SERVER_SSL_ENABLED="true"
- SERVER_SSL_KEY=/usr/share/kibana/config/certs/privkey.pem
- SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/fullchain.pem
- SERVER_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/chain.pem
${MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://dev.mysite.com:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
volumes:
esdata01:
driver: local
esdata02:
driver: local
esdata03:
driver: local
kibanadata:
driver: local
Now everything should work fine!