In my Kubernetes cluster, Rancher never creates Persistent Volumes after creating a Persistent Volume Claim and applying a Pod.
Solution/Work around available under second update.
The cluster has been installed with Kubespray. The configuration for local path provisioning in inventory/myclster/group_vars/k8s-cluster/addons.yml:
# Rancher Local Path Provisioner
local_path_provisioner_enabled: true
# local_path_provisioner_namespace: "local-path-storage"
# local_path_provisioner_storage_class: "local-path"
# local_path_provisioner_reclaim_policy: Delete
# local_path_provisioner_claim_root: /opt/local-path-provisioner/
# local_path_provisioner_debug: false
# local_path_provisioner_image_repo: "rancher/local-path-provisioner"
# local_path_provisioner_image_tag: "v0.0.14"
# local_path_provisioner_helper_image_repo: "busybox"
# local_path_provisioner_helper_image_tag: "latest"
# Local volume provisioner deployment
local_volume_provisioner_enabled: false
# local_volume_provisioner_namespace: kube-system
# local_volume_provisioner_nodelabels:
# - kubernetes.io/hostname
# - topology.kubernetes.io/region
# - topology.kubernetes.io/zone
# local_volume_provisioner_storage_classes:
# local-storage:
# host_dir: /mnt/disks
# mount_dir: /mnt/disks
# volume_mode: Filesystem
# fs_type: ext4
# fast-disks:
# host_dir: /mnt/fast-disks
# mount_dir: /mnt/fast-disks
# block_cleaner_command:
# - "/scripts/shred.sh"
# - "2"
# volume_mode: Filesystem
# fs_type: ext4
Steps to recreate the problem:
Create PVC:
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml
Result:
Created PVC, name local-path-pvc, status Pending, storage class local-path
Create Pod
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
Result:
Created Pod, name create-pvc-123, status Waiting:ContainerCreating.
Describing Pod with kubectl:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> error while running "VolumeBinding" prebind plugin for pod "create-pvc-123": Failed to bind volumes: timed out waiting for the condition
I have tried different charts, and in all cases no Persistent Volume has been created. The ServiceAccount local-path-provisioner-service-account exists. The Deployment of the local path provisioner has one Pod.
UPDATE
On the server the logs contains several errors, sudo journalctl -xeu kubelet | grep 'fail':
...
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.000246 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script podName:71b44438-fadb-4859-a788-8d911dfab2db nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.000164134 +0200 CEST m=+9380.643933974 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script\") pod \"create-pvc-80d115d9-98fd-4fcd-9e41-55b74f809efb\" (UID: \"71b44438-fadb-4859-a788-8d911dfab2db\") : configmap references non-existent config key: setup"
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.404015 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.403980548 +0200 CEST m=+9381.047750378 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"
Oct 12 16:54:40 node1 kubelet[274306]: E1012 16:54:40.464999 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:56:42.464936126 +0200 CEST m=+9503.108706016 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"
UPDATE - solution?
I changed the ConfigMap 'local-path-config' as described in the docs.
However, the jinja template in Kubespray lacks the properties 'setup' and 'teardown' in the configuration.
When I added de setup and teardown properties Kubernetes created the PV and the Pod started.
What's the reason Kubespray doesn't provide these properties in the template?
Default version that kubespray should use is local_path_provisioner_image_tag: "v0.0.14". I suppose the source for the template is https://github.com/rancher/local-path-provisioner/blob/v0.0.14/deploy/example-config.yaml. It doesn't have setup and teardown properties, they were introduced in v0.0.15.