I have an nginx web server deployed with docker swarm, and I want to be able to deploy it also with kubernetes.
Right now I'm having trouble inserting the nginx configuration files into the container.
I'll first post here what I already do in docker swarm, and then what I tried in kubernetes.
Dockerfile:
FROM "nginx:1.19.1-alpine" AS nginx
[...]
RUN \
rm -fv /etc/nginx/nginx.conf && \
ln -svT /usr/local/etc/nginx/nginx.conf \
/etc/nginx/nginx.conf && \
rm -frv /etc/nginx/conf.d && \
ln -svT /usr/local/etc/nginx/conf.d \
/etc/nginx/conf.d
[...]
Basically I set up the image so that I can place my custom nginx config files into /usr/local/etc/ instead of /etc/
Docker swarm:
docker-compose.yaml:
configs:
nginx_conf:
file: /run/configs/www/etc/nginx/nginx.conf
nginx_conf_security-parameters:
file: /run/configs/www/etc/nginx/conf.d/security-parameters.conf
nginx_conf_server:
file: /run/configs/www/etc/nginx/conf.d/server.conf
networks:
alejandro-colomar:
services:
www:
configs:
-
mode: 0440
source: nginx_conf
target: /usr/local/etc/nginx/nginx.conf
-
mode: 0440
source: nginx_conf_security-parameters
target: /usr/local/etc/nginx/conf.d/security-parameters.conf
-
mode: 0440
source: nginx_conf_server
target: /usr/local/etc/nginx/conf.d/server.conf
deploy:
placement:
constraints:
- node.role == worker
replicas: 1
restart_policy:
condition: any
image: "alejandrocolomar/www:0.16-a6-kube"
networks:
-
"alejandro-colomar"
ports:
- "32001:8080"
version: "3.8"
Here I take the nginx custom config files, which I first put in /run/configs/ with a script so that they are in ram, and introduce them in the container as configs in the right place (/usr/local/etc/nginx/ and its subdir conf.d/).
I'd like to do the same in kubernetes, and I read that I should use a projected volume for that (if it's doable with normal volumes or in some other way, without having to use any dirty workarounds, I'm also open to that), so I tried the following (after seeing some examples which weren't very clear to me):
config.sh:
kubectl create configmap "nginx-conf-cm" \
--from-file "/run/configs/www/etc/nginx/nginx.conf"
kubectl create configmap "nginx-conf-security-parameters-cm" \
--from-file "/run/configs/www/etc/nginx/conf.d/security-parameters.conf"
kubectl create configmap "nginx-conf-server-cm" \
--from-file "/run/configs/www/etc/nginx/conf.d/server.conf"
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: www-deploy
spec:
replicas: 1
selector:
matchLabels:
service: www-svc
template:
metadata:
labels:
service: www-svc
spec:
containers:
-
image: "alejandrocolomar/www:0.16-a6-kube"
name: www-container
volumeMounts:
-
mountPath: /usr/local/etc/nginx/
name: nginx-volume
readOnly: true
volumes:
-
name: nginx-volume
projected:
sources:
-
configMap:
name: nginx-conf-cm
path: "nginx.conf"
-
configMap:
name: nginx-conf-security-parameters-cm
path: "conf.d/security-parameters.conf"
-
configMap:
name: nginx-conf-server-cm
path: "conf.d/server.conf"
service.yaml: (This one I put it here only for completeness)
apiVersion: v1
kind: Service
metadata:
name: www
spec:
ports:
-
nodePort: 32001
port: 8080
selector:
service: www-svc
type: NodePort
The deployment failed, of course, but it wasn't very wrong I think. When I entered in the container to debug it, the problem was that the three files were placed into /usr/local/etc/nginx/, and the subdir conf.d/ was not created:
/usr/local/etc/nginx/nginx.conf
/usr/local/etc/nginx/security-parameters.conf
/usr/local/etc/nginx/server.conf
How should I fix that (presumably in deployment.yaml) so that I have the following files in the container?:
/usr/local/etc/nginx/nginx.conf
/usr/local/etc/nginx/conf.d/security-parameters.conf
/usr/local/etc/nginx/conf.d/server.conf
There are multiple ways to handle this. One solution would be to create 3 separate volumes from these individual config maps and mount each one into its appropriate destination file/folder.
volumes:
- name: nginx-cm
configMap:
name: nginx-conf-cm
- name: nginx-sec
configMap:
name: nginx-conf-security-parameters-cm
- name: nginx-serv
configMap:
name: nginx-conf-server-cm
...
containers:
-
image: "alejandrocolomar/www:0.16-a6-kube"
name: www-container
volumeMounts:
- mountPath: /usr/local/etc/nginx/nginx.conf
name: nginx-cm
subPath: nginx.conf
readOnly: true
- mountPath: /usr/local/etc/nginx/conf.d/security-parameters.conf
name: nginx-sec
subPath: security-parameters.conf
readOnly: true
- mountPath: /usr/local/etc/nginx/conf.d/server.conf
name: nginx-serv
subPath: server.conf
readOnly: true
using mountPath and subPath in volumeMounts allows you to pick specific file from a given config map (doesn't matter much since you have on file per cm) and mount it as a file (not overriding the other content in the existing folder).
To explain a bit the code above:
- mountPath: /usr/local/etc/nginx/nginx.conf
name: nginx-cm
subPath: nginx.conf
tells kubernetes to use volume with the name of nignx-cm (defined in a volume section). Pick file nginx.conf (via subpath) that can be found in the associated config map and expose it in the container (mountPath) at the location /usr/local/etc/nginx/nginx.conf.
I have not run the code so there might be typos
PS: note that it might be better to create a custom configuration file inside of ../etc/nginx/conf.d/ instead of overriding ../etc/nginx/nginx.conf. That way you wouldn't need to worry about destroying the original files ../etc/nginx/ and you could mount the whole volume instead of using subpath (to avoid issues with config updates)