I am trying to renew a clusters set of certificates that are stored on an ami and used to boot up a Kubernetes cluster, currently if the certs get to over a year old the cluster can not be started. Using kubeadm is not a solution due to the added start up time to the ami and therefore I am trying to have the certificates have a 10 year expiry on the ami at creation (I am not worried about the security risks to due this being an nearly air gaped training environment).
Currently I have tried this method to extend the expiry time which works, it allows minikube to start however I am still facing issues with the api server:
# Extend Certificates
extend_certificate() {
local base_path="$1"
local node_name="$2"
local days=3649
cd "$base_path"
# Map of COMMON_NAME to "KEY_NAME:CA_CERT:CA_KEY"
declare -A map=(
["front-proxy-client"]="front-proxy-client.key:front-proxy-ca.crt:front-proxy-ca.key"
["apiserver-etcd-client"]="apiserver-etcd-client.key:etcd/ca.crt:etcd/ca.key"
["apiserver-kubelet-client"]="apiserver-kubelet-client.key:ca.crt:ca.key"
)
for COMMON_NAME in "${!map[@]}"; do
IFS=":" read -r KEY_NAME CA_CERT CA_KEY <<< "${map[$COMMON_NAME]}"
openssl11 req -new -key "$KEY_NAME" -out "$COMMON_NAME.csr" -subj "/CN=$COMMON_NAME"
openssl11 x509 -req -in "$COMMON_NAME.csr" -CA "$CA_CERT" -CAkey "$CA_KEY" -CAcreateserial -out "${COMMON_NAME}_new.crt" -days $days
mv "$COMMON_NAME.crt" "${COMMON_NAME}_old.crt"
mv "${COMMON_NAME}_new.crt" "$COMMON_NAME.crt"
done
cd "$base_path/etcd"
COMMON_NAME="kube-etcd-healthcheck-client"
openssl11 req -new -key healthcheck-client.key -out healthcheck-client.csr -subj "/O=system:masters/CN=$COMMON_NAME"
openssl11 x509 -req -in healthcheck-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out healthcheck-client_new.crt -days $days
mv healthcheck-client.crt healthcheck-client_old.crt
mv healthcheck-client_new.crt healthcheck-client.crt
COMMON_NAME="$node_name"
for cert_name in peer server; do
openssl11 req -new -key "$cert_name.key" -out "$cert_name.csr" -subj "/CN=$COMMON_NAME" -addext "subjectAltName = DNS:localhost, DNS:$node_name, IP:192.168.49.2, IP:127.0.0.1, IP:0:0:0:0:0:0:0:1"
openssl11 x509 -req -in "$cert_name.csr" -CA ca.crt -CAkey ca.key -CAcreateserial -out "${cert_name}_new.crt" -days $days -extensions SAN -extfile <(printf "\n[SAN]\nsubjectAltName=DNS:localhost, DNS:$node_name, IP:192.168.49.2, IP:127.0.0.1, IP:0:0:0:0:0:0:0:1")
mv "$cert_name.crt" "${cert_name}_old.crt"
mv "${cert_name}_new.crt" "$cert_name.crt"
done
}
extend_certificate "/var/lib/docker/volumes/sf-node-1/_data/lib/minikube/certs" "$NODE_NAME"
extend_certificate "/var/lib/docker/volumes/sfdev-node-1/_data/lib/minikube/certs" "$DEV_NODE_NAME"
Which causes new RBAC errors such as, these are not present if I do not touch the certs:
++ kubectl get svc -n ingress-nginx
++ grep -oP '^ingress-nginx-.*controller\b'
++ awk 'NR==1{print $1}'
error: error upgrading connection: unable to upgrade connection: Forbidden (user=apiserver-kubelet-client, verb=create, resource=nodes, subresource=proxy)
As you can see here the certificates do seem to be correct:
[root@ip-xxx-xx-xx-xxx xxxxxx]$ kubeadm certs check-expiration --cert-dir /var/lib/docker/volumes/sf-node-1/_data/lib/minikube/certs
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
!MISSING! admin.conf
apiserver Aug 19, 2033 14:21 UTC 9y ca no
apiserver-etcd-client Aug 18, 2033 14:25 UTC 9y etcd-ca no
apiserver-kubelet-client Aug 18, 2033 14:25 UTC 9y ca no
!MISSING! controller-manager.conf
etcd-healthcheck-client Aug 18, 2033 14:25 UTC 9y etcd-ca no
etcd-peer Aug 18, 2033 14:25 UTC 9y etcd-ca no
etcd-server Aug 18, 2033 14:25 UTC 9y etcd-ca no
front-proxy-client Aug 18, 2033 14:25 UTC 9y front-proxy-ca no
!MISSING! scheduler.conf
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 19, 2033 14:21 UTC 9y no
etcd-ca Aug 19, 2033 14:21 UTC 9y no
front-proxy-ca Aug 19, 2033 14:21 UTC 9y no
For anyone else looking to extend minikubes certs in this way, note that the extend function was missing an organisation for the two apiserver certs.
The correct version is:
extend_certificate() {
local base_path="$1"
local node_name="$2"
local days=3649
cd "$base_path"
# Map of COMMON_NAME to "KEY_NAME:CA_CERT:CA_KEY"
declare -A map=(
["kube-apiserver-etcd-client"]="apiserver-etcd-client.key:etcd/ca.crt:etcd/ca.key"
["kube-apiserver-kubelet-client"]="apiserver-kubelet-client.key:ca.crt:ca.key"
)
for COMMON_NAME in "${!map[@]}"; do
IFS=":" read -r KEY_NAME CA_CERT CA_KEY <<< "${map[$COMMON_NAME]}"
openssl11 req -new -key "$KEY_NAME" -out "$KEY_NAME.csr" -subj "/O=system:masters/CN=$COMMON_NAME"
openssl11 x509 -req -in "$KEY_NAME.csr" -CA "$CA_CERT" -CAkey "$CA_KEY" -CAcreateserial -out "${KEY_NAME}_new.crt" -days $days
mv "$KEY_NAME.crt" "${KEY_NAME}_old.crt"
mv "${KEY_NAME}_new.crt" "$KEY_NAME.crt"
done
COMMON_NAME="front-proxy-client"
openssl11 req -new -key front-proxy-client.key -out front-proxy-client.csr -subj "/CN=$COMMON_NAME"
openssl11 x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key -CAcreateserial -out front-proxy-client_new.crt -days $days
mv front-proxy-client.crt front-proxy-client_old.crt
mv front-proxy-client_new.crt front-proxy-client.crt
cd "$base_path/etcd"
COMMON_NAME="kube-etcd-healthcheck-client"
openssl11 req -new -key healthcheck-client.key -out healthcheck-client.csr -subj "/O=system:masters/CN=$COMMON_NAME"
openssl11 x509 -req -in healthcheck-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out healthcheck-client_new.crt -days $days
mv healthcheck-client.crt healthcheck-client_old.crt
mv healthcheck-client_new.crt healthcheck-client.crt
COMMON_NAME="$node_name"
for cert_name in peer server; do
openssl11 req -new -key "$cert_name.key" -out "$cert_name.csr" -subj "/CN=$COMMON_NAME" -addext "subjectAltName = DNS:localhost, DNS:$node_name, IP:192.168.49.2, IP:127.0.0.1, IP:0:0:0:0:0:0:0:1"
openssl11 x509 -req -in "$cert_name.csr" -CA ca.crt -CAkey ca.key -CAcreateserial -out "${cert_name}_new.crt" -days $days -extensions SAN -extfile <(printf "\n[SAN]\nsubjectAltName=DNS:localhost, DNS:$node_name, IP:192.168.49.2, IP:127.0.0.1, IP:0:0:0:0:0:0:0:1")
mv "$cert_name.crt" "${cert_name}_old.crt"
mv "${cert_name}_new.crt" "$cert_name.crt"
done
}