Search code examples
amazon-web-serviceskuberneteskubeadm

kubeadm init phase upload-config failing

I am new to kubernetes and want to setup kubernetes HA setup after successfully completing examples with minikube and single master kubernetes cluster. I am using AWS EC2 instances and AWS application load balancer for this purpose. I dont want to use KOPS or any other tool for installation. I want to get hands on with kubeadm. I followed below steps

  1. Created self signed certificate ca.crt and ca.key to use for kubernetes
  2. Installed this certificate as Root CA on my ubuntu instance
  3. Copied this ca.crt and ca.key to /etc/kubernetes/pki
  4. Created new certificate for aws loadbalancer and signed it with above ca.crt. With this certificate created Aws application load balancer
  5. I also created Record Set in AWS Route53 for domain name mapping. I also made sure this domain name mapping is working. (i.e. master.k8sonaws.com is properly resolving to aws load balancer)

  6. Now I am using kubeadm init 

    kubeadm init --pod-network-cidr=192.168.0.0/20
--service-cidr=192.168.16.0/20 --node-name=10.0.0.13  --control-plane-endpoint "master.k8sonaws.com:443"  --upload-certs --v=8 --apiserver-bind-port=443 --apiserver-cert-extra-sans=master.k8sonaws.com,i-0836dd4dc6609a924

  7. This command is succeeding up-to upload-config phase. Health check endpoint is returning success but after that its failing in upload-config phase

    configmaps is forbidden: User "system:anonymous" cannot create resource "configmaps" in API group "" in the namespace "kube-system"

Here I am not able to understand why kubeadm is passing anonymyous user for api call. How can I resolve this issue

Solution

  • I found solution to this problem after trying lot of different things for 2 days. Problem is aws load balancer does not pass client certificate to backend server when using https listener. And problem is Aws has not documented this fact or I did not get those document if they are there.

    Way to solve this problem is using plane http listener using same port 443. So SSL termination is carried out by backened server. In my case this is not security threat since both my load balancer and backend servers are internal and not exposed to public ip.