I'm trying to setup a Kubernetes cluster in Rackspace, and I understand that to get persistent volume support I would need to use Cinder (Openstack supported by Rackspace).
Following the Cloud Provider Integrations setup guide, I have setup /etc/kubernetes/cloud-config as follows
[Global]
username=cinder
password=********
auth-url=https://identity.api.rackspacecloud.com/v2.0
tenant-name=1234567
region=LON
I've added the following to the kubelet startup command in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
--cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud-config
And I'm then running kubeadm init --config=kubeadm.conf where kubeadm.conf is:
kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
cloudProvider: openstack
pod-network-cidr: 10.244.0.0/16
It fails waiting for the kubelet to start. I tracked down the kubelet error to the following:
07:24:51.407692 21412 feature_gate.go:156] feature gates: map[]
07:24:51.407790 21412 controller.go:114] kubelet config controller: starting controller
07:24:51.407849 21412 controller.go:118] kubelet config controller: validating combination of defaults and flags
07:24:51.413973 21412 mount_linux.go:168] Detected OS with systemd
07:24:51.414065 21412 client.go:75] Connecting to docker on unix:///var/run/docker.sock
07:24:51.414137 21412 client.go:95] Start docker client with request timeout=2m0s
07:24:51.415471 21412 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
07:24:51.437924 21412 iptables.go:564] couldn't get iptables-restore version; assuming it doesn't support --wait
07:24:51.440245 21412 feature_gate.go:156] feature gates: map[]
07:24:52.066765 21412 server.go:301] Successfully initialized cloud provider: "openstack" from the config file: "/etc/kubernetes/cloud-config"
07:24:52.066984 21412 openstack_instances.go:39] openstack.Instances() called
07:24:52.067048 21412 openstack_instances.go:46] Claiming to support Instances
07:24:52.070870 21412 metadata.go:84] Unable to run blkid: exit status 2
07:24:52.070993 21412 metadata.go:124] Attempting to fetch metadata from http://169.254.169.254/openstack/2012-08-10/meta_data.json
07:25:22.071444 21412 metadata.go:127] Cannot read http://169.254.169.254/openstack/2012-08-10/meta_data.json: Get http://169.254.169.254/openstack/2012-08-10/meta_data.json: dial tcp 169.254.169.254:80: i/o timeout
error: failed to run Kubelet: error fetching current node name from cloud provider: Get http://169.254.169.254/openstack/2012-08-10/meta_data.json: dial tcp 169.254.169.254:80: i/o timeout
How can I debug this further? I don't really understand how the IP address 169.254.169.254 works in this request.
Right now I can't tell if I have a Kubernetes issue or a Rackspace issue.
The answer is that Rackspace Cloud does not use the Openstack Metadata service. Instead it uses cloud-init with config-drive - a read-only block device (virtual CD-ROM) that is attached at boot.
The config drive contains the cloud-init data. Example https://developer.rackspace.com/blog/using-cloud-init-with-rackspace-cloud/
Anecdotally it seems most Rackspace customers who are using Kubernetes use CoreOS VMs which support cloud-config and the Openstack config drive. When K8s runs on a machine with the drive mounted, it attempts to obtain the metadata from there.