Search code examples
kubernetesibm-cloud

IBM File Storage on Kubernetes stuck "Pending"


I am trying to use the following https://cloud.ibm.com/docs/containers?topic=containers-file_storage#add_file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ibmc-file
  labels:
    billingType: 'monthly'
    region: us-south
    zone: dal10
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 12Gi
  storageClassName: ibmc-file-silver
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:11
          imagePullPolicy: Always
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - name: postgres-storage
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-storage
          persistentVolumeClaim:
            claimName: ibmc-file

But the PVC is never "Bound" and gets stuck as "Pending".

➜  postgres-kubernetes kubectl describe pvc ibmc-file
Name:          ibmc-file
Namespace:     default
StorageClass:  ibmc-file-silver
Status:        Pending
Volume:        
Labels:        billingType=monthly
               region=us-south
               zone=dal10
Annotations:   ibm.io/provisioning-status=failed: Storage creation failed with error: {Code:E0013, Description:User doesn't have permissions to create or manage Storage [Backend Error:Validation failed due to missin...
               kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"billingType":"monthly","region":"us-south","zone":"dal10"},"n...
               volume.beta.kubernetes.io/storage-provisioner=ibm.io/ibmc-file
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason                Age                From                                                                                    Message
  ----     ------                ----               ----                                                                                    -------
  Normal   Provisioning          10m (x3 over 10m)  ibm.io/ibmc-file_ibm-file-plugin-5d7684d8c5-xlvks_db50c480-500f-11e9-ba08-cae91657b92d  External provisioner is provisioning volume for claim "default/ibmc-file"
  Warning  ProvisioningFailed    10m (x3 over 10m)  ibm.io/ibmc-file_ibm-file-plugin-5d7684d8c5-xlvks_db50c480-500f-11e9-ba08-cae91657b92d  failed to provision volume with StorageClass "ibmc-file-silver": Storage creation failed with error: {Code:E0013, Description:User doesn't have permissions to create or manage Storage [Backend Error:Validation failed due to missing permissions[NAS_MANAGE] for User[id:xxx, name:xxxm_2018-11-20-07.35.49, email:xxx, account:xxx]], Type:MissingStoragePermissions, RC:401, Recommended Action(s):Run `ibmcloud ks api-key-info` to see the owner of the API key that is used to order storage. Then, contact the account administrator to add the missing storage permissions. If infrastructure credentials were manually set via `ibmcloud ks credentials-set`, check the permissions of that user. Delete the PVC and re-create it. If the problem persists, open an IBM Cloud support case.}
  Normal   ExternalProvisioning  7m (x22 over 10m)  persistentvolume-controller                                                             waiting for a volume to be created, either by external provisioner "ibm.io/ibmc-file" or manually created by system administrator
  Normal   ExternalProvisioning  11s (x26 over 6m)  persistentvolume-controller                                                             waiting for a volume to be created, either by external provisioner "ibm.io/ibmc-file" or manually created by system administrator

Solution

  • @atkayla Could you try running kubectl get secret storage-secret-store -n kube-system -o yaml | grep slclient.toml: | awk '{print $2}' | base64 --decode to see what API key is used in the storage secret store? If this also shows your name and email address, then the file storage plug-in uses the permissions that are assigned to you.

    You might have the permissions to create the cluster, but you might lack some storage permissions that do not let you create the storage. Are you the owner of the account and have the possibility to check the permissions? You should have Add/Upgrade Storage (StorageLayer), and Storage Manage.

    If you do not have these permissions, add these and then run ibmcloud ks api-key-set to update the API key. The storage secret store is automatically refreshed after 5-15 minutes. Then, you can try again.