Insecure Docker registry and self-signed certificates

Question

I have already setup a localized docker registry with self-signed certificates based on this reference: https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry.

I pussed an example image using:

sudo docker run hello-world
sudo docker tag hello-world registry.local.doc:5045/hello-world
sudo docker push registry.local.doc:5045/hello-world

I am able to see the "contents" of the registry on https://registry.local.doc:5045/v2/_catalog:

{"repositories":["hello-world"]}

1st Problem

I have used self-signed certificates using:

openssl req -newkey rsa:4096 -nodes -sha256 -keyout ./certs/tls.key -x509 -days 365  -subj "/C=GR/ST=./L=./O=./CN=registry.local.doc" -addext "subjectAltName = DNS:registry.local.doc" -out ./certs/tls.crt

However, in order to make it work I have to add in /etc/docker/daemon.json the following:

{ "insecure-registries" : ["registry.local.doc:5000"] }

which is confusing as I am using self-signed certificates as already mentioned.

2nd Problem

I have a k3s node and I want to pull images from inside a Pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-test
  labels:
    app: registry-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry-test
  template:
    metadata:
      labels:
        app: registry-test
    spec:
      containers:
      - name: registry-test
        image: registry.local.doc:5045/hello-world

It works but first I need to run:

sudo update-ca-certificates
sudo systemctl restart containerd
sudo systemctl restart k3s 

I understand the reasons behind the execution of the first two commnads sudo update-ca-certificates & sudo systemctl restart containerd, however I have no clue why I have to run sudo systemctl restart k3s to make it works.

Is there a solution without restarting the cluster, as something like this is forbidden when we talk about a cluster in production?

And if I had a k8s cluster, what exactly would Ι have to restart?

Answer 1

1st Problem registry is a server side, your docker is client side, the config insecure-registries tell your docker to skip server cert validation.
Without this settings, docker will not pull image because the cert is invalid.

2st Problem you need to restart k3s only because you using insecure-registries config. docker need restart to reload this config.

When you using self-sign cert registry, you have two way to pull image from there.

1. Add { "insecure-registries" : ["registry.local.doc:5000"] } to /etc/docker/daemon.json and restart k3s

   sudo systemctl restart containerd
   sudo systemctl restart k3s 
   

   In production, normally you should have more then one worker and follow below step:

   1. drain the worker, see doc here. This will move your pod to other worker.
   2. add insecure-registries and restart k3s.
   3. uncordon the worker, pod can move back to this worker.
   4. repeat above step on every worker.

2. Add your self-sign cert to trust store on every woker. This way don't require restart k3s. For ubuntu see below.

   sudo cp tls.crt /usr/local/share/ca-certificates
   sudo update-ca-certificates
   

3. add cert to docker certs folder

   Instruct every Docker daemon to trust that certificate. The way to do this depends on your OS.
   Linux: Copy the domain.crt file to /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt on every Docker host. You do not need to restart Docker.

   sudo cp tls.crt /etc/docker/certs.d/registry.local.doc:5000/ca.crt