I tried to deploy Keycloak in production mode. These are the Dockerfile, docker build command and docker run command that I have used.
before deploying keycloak, I have created the docker network and mysql database.
1. create network:
docker network create keycloak-network
2. deploy kc_mysql db (a mysql database):
docker run -v /var/lib/docker/volumes/kc_mysql:/var/lib/mysql -p 3307:3306 --name kc_mysql -d --net keycloak-network -e MYSQL_DATABASE=keycloak -e MYSQL_USER=keycloak -e MYSQL_PASSWORD=xxxx -e MYSQL_ROOT_PASSWORD=yyyy mysql
Then I built docker image and run the container,
3. Dockerfile:
# Use the builder image to build with the necessary configurations
FROM quay.io/keycloak/keycloak:latest as builder
# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange
# Configure a database vendor
ENV KC_DB=mysql
WORKDIR /opt/keycloak
# Build the configuration
RUN /opt/keycloak/bin/kc.sh build --cache=ispn
# Final image
FROM quay.io/keycloak/keycloak:latest
# Copy the built artifacts from the builder stage
COPY --from=builder /opt/keycloak/ /opt/keycloak/
ENV KC_DB=mysql
ENV KC_DB_URL=jdbc:mysql://kc_mysql:3306/keycloak
ENV KC_DB_USERNAME=yyyy
ENV KC_DB_PASSWORD=xxxx
ENV KC_HOSTNAME=localhost
# Set the entry point to start Keycloak
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
4. Docker build command
docker build . -t mykeycloak
5. Docker run command
docker run -v /var/lib/docker/volumes/keycloak:/opt/keycloak/conf -v /etc/letsencrypt/live/abc.xyz.com:/etc/x509/https -e KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/privkey.pem -e KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/fullchain.pem --name mykeycloak -p 8443:8443 -d --net keycloak-network -e KEYCLOAK_ADMIN=admin-e KEYCLOAK_ADMIN_PASSWORD=admin mykeycloak start --optimized
The SSL certificates are used. But I get the below logs when I Run the container. At the bottom of logs I got the error related to SSL certificates.
INFO [org.keycloak.common.Profile] (main) Preview features enabled: token-exchange
INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: localhost, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 73276fc8-8609-4db5-a34a-73d73e92dbb1, name: 768a15960a95-45041
2023-11-14 04:30:29,649 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) 768a15960a95-45041: no members discovered after 2004 ms: creating cluster as coordinator
INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [768a15960a95-45041|0] (1) [768a15960a95-45041]
INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `768a15960a95-45041`, physical addresses are `[172.22.0.3:48009]`
WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: 768a15960a95-45041, Site name: null
INFO [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: /etc/x509/https/fullchain.pem
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Key material not provided to setup HTTPS. Please configure your keys/certificates or start the server in development mode.
ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
As I see in many places, I changed the permissions for certificates location /etc/letsencrypt/live/abc.xyz.com
the certificates are;
drwxr-xr-x. 2 root user1 93 Nov 9 13:47 .
drwx--x--x. 3 root root 48 Nov 9 13:47 ..
lrwxrwxrwx. 1 root user1 44 Nov 9 13:47 cert.pem -> ../../archive/abc.xyz.com/cert1.pem
lrwxrwxrwx. 1 root user1 45 Nov 9 13:47 chain.pem -> ../../archive/abc.xyz.com/chain1.pem
lrwxrwxrwx. 1 root user1 49 Nov 9 13:47 fullchain.pem -> ../../archive/abc.xyz.lk/fullchain1.pem
lrwxrwxrwx. 1 root user1 47 Nov 9 13:47 privkey.pem -> ../../archive/abc.xyz.com/privkey1.pem
-rw-r--r--. 1 root user1 692 Nov 9 13:47 README
I am the user1, but there is another user user2. now I checked the permission for certificates for root, user1 and user2. Permissions have no problem.
sudo -u user1 file ../../archive/abc.xyz.com/fullchain1.pem
../../archive/abc.xyz.com/fullchain1.pem: PEM certificate
sudo -u user2 file ../../archive/abc.xyz.com/fullchain1.pem
../../archive/abc.xyz.com/fullchain1.pem: PEM certificate
But the error exists. My all previous docker images have root user in when I check using docker inspect <image_name>.
But for this keycloak custom docker image, the user is 1000 (for user2).
"Config": {
"Hostname": "768a15960a95",
"Domainname": "",
"User": "1000",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8080/tcp": {},
"8443/tcp": {}
},
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/docker/volumes/keycloak",
"Destination": "/opt/keycloak/conf",
"Mode": "",
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Source": "/etc/letsencrypt/live/abc.xyz.com",
"Destination": "/etc/x509/https",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
In host;
grep 1000 /etc/passwd
>>user2
grep 1001 /etc/passwd
>>user1
I tried the keyword USER in Dockerfile.
USER root or USER 1001
But it didn't change the user in image's or container's user.
I am the user1 (1001) who built and run the keycloak container, but thisis unusual and can someone explain what happened here and although the permissions are set to certificates files in host, why the keycloak container does not up in production mode. How can i run keycloak avoiding the error (main) ERROR: /etc/x509/https/fullchain.pem. Without using certificates, the container could be run in development mode without problem using start-dev. But i need the production mode deployment.
After few days I could find an answer for this issue.
vendor: quay.io/keycloak/keycloak
version: Keycloak version is 22.0.5
First I moved to docker-compose;
version: '3'
services:
mykeycloak:
build:
context: .
dockerfile: Dockerfile
image: my-keycloak:latest
container_name: mykeycloak
environment:
- KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/privkey.pem
- KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/fullchain.pem
- KC_DB=mysql
- KC_DB_URL=jdbc:mysql://kc_mysql:3306/keycloak
- KC_DB_USERNAME=<kc_username>
- KC_DB_PASSWORD=<kc_password>
- KC_HOSTNAME=abc.xyz.com
- KC_HOSTNAME_PORT=8443
- KC_HEALTH_ENABLED=true
- KC_METRICS_ENABLED=true
- KC_FEATURES=token-exchange
- KEYCLOAK_ADMIN=<admin>
- KEYCLOAK_ADMIN_PASSWORD=<admin_password>
volumes:
- /var/lib/docker/volumes/keycloak:/opt/keycloak/conf
- /etc/letsencrypt/live/abc.xyz.com/fullchain.pem:/etc/x509/https/fullchain.pem
- /etc/letsencrypt/live/abc.xyz.com/privkey.pem:/etc/x509/https/privkey.pem
ports:
- "8443:8443"
networks:
- keycloak-network
command: ["start"]
networks:
keycloak-network:
external: true
Here, I've changed HOSTNAME from localhost to my real domain name abc.xyz.com.
I get the same error as above my question. The problem seems to be a permission problem on my host. Docker user does not have proper permissions to use fullchain.pem. Now tried this command on my host.
Now, following is the solving steps:
Keycloak in development modeFor this I had to comment below two environment variables on docker-compose as below:
- KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/privkey.pem
- KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/fullchain.pem
Then changed command from start to start-dev as below:
command: ["start-dev"]
Then the container is running and I checked whether it is a permission issue or not, using a cat command to the /etc/x509/https/fullchain.pem file as below..
docker exec -it <keycloak_container_id> cat /etc/x509/https/fullchain.pem
response:
cat: /etc/x509/https/fullchain.pem: Permission denied
Now, it was clear that the issue is a permission problem.
Then I execute the below command and solved the permission problem.
chmod -R 644 /etc/letsencrypt/live/abc.xyz.com/fullchain.pem
Again I try the same command in step (2)
docker exec -it <keycloak_container_id> cat /etc/x509/https/fullchain.pem
response:
-----BEGIN CERTIFICATE-----
MIIGLTCCBRWgAwIBAgIRAMvuT68080185tVdPX+AoIAwDQYJKoZIhvcNAQELBQAw
Problem solved!!. Now execute docker-compose down and uncomment the above commented environment variables and change command from start-dev to start or start --optimized
Now execute the docker-compose up. Now Keycloak is running in production mode successfully.
check the logs using docker logs -f <keycloak_container_id>
INFO [io.quarkus] (main) Keycloak 22.0.5 on JVM (powered by Quarkus 3.2.7.Final) started in 7.837s. Listening on: https://0.0.0.0:8443 INFO [io.quarkus] (main) Profile prod activated. INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, vertx]
Done!! Now you are able to access Keycloak admin panel via https://abc.xyz.com:8443