Can not access my kubernetes cluster even if all my server certificates are valid

I am trying to achieve that my kubernetes cluster should have a validity of 5 years, so I have made my ca.crt, apiserver.crt, kubelet-client.crt, front-proxy.crt of 5 years validity and placed those in /etc/kubernetes/pki.

Also, I have enabled my kubelet with client certificate rotation

Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --feature-gates=RotateKubeletClientCertificate=true"

So to verify my cluster is working fine I changed the date on my system to 1 day before 1 year expiration and certificate rotation are done properly

Oct 22 06:00:16 ip-10-0-1-170.ec2.internal kubelet[28887]: I1022 06:00:16.806115   28887 reconciler.go:154] Reconciler: start to sync state
Oct 22 06:00:23 ip-10-0-1-170.ec2.internal kubelet[28887]: I1022 06:00:23.546154   28887 transport.go:126] certificate rotation detected, shutting down client connections to start using new credentials

But once my cluster passes one year it starts showing the error on any kubectl get nodes/pods command: "error: You must be logged in to the server (Unauthorized)"

The possible issue I can think is /etc/kubernetes/admin.conf has only one-year validity certificates. Thanks for your help

Solution

I have figured out a way to regenerate new admin.conf certificate before expiry of cluster

Generate admin.key and admin.csr using openssl

openssl genrsa -out admin.key 2048 
openssl req -new -key admin.key -out admin.csr -subj "/O=system:masters/CN=kubernetes-admin"

Now create CSR in kubernetes using above openssl admin.csr

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: admin_csr
spec:
  groups:
  - system:authenticated
  request: $(cat admin.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - client auth
EOF

Now approve the CSR generated using kubectl certificate approve admin_csr

Now extract the admin.crt from approved CSR kubectl get csr admin_csr -o jsonpath='{.status.certificate}' | base64 -d > admin.crt

Now change the current user and context to use the new admin key and certificates.

kubectl config set-credentials kubernetes-admin --client-certificate=/home/centos/certs/admin.crt  --client-key=/home/centos/certs/admin.key
kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin

After this step your kubeconfig which in my case is /root/.kube/config has new client certificate data and key.

Hope this helps.