Search code examples
elasticsearchkubernetespersistencekubernetes-pvc

PVCs not created at all after deletion, when using Retail reclaim policy in corresponding StorageClass

I am using the ECK operator, to create an Elasticsearch instance.

The instance uses a StorageClass that has Retain (instead of Delete) as its reclaim policy.

Here are my PVCs before deleting the Elasticsearch instance

▶ k get pvc
NAME                                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
elasticsearch-data--es-multirolenodes1-0   Bound    pvc-ba157213-67cf-4b81-8fe2-6211b771e62c   20Gi       RWO               balanced-retain-csi   8m15s
elasticsearch-data--es-multirolenodes1-1   Bound    pvc-e77dbb00-7cad-419f-953e-f3398e3860f4   20Gi       RWO               balanced-retain-csi   7m11s
elasticsearch-data--es-multirolenodes1-2   Bound    pvc-b258821b-0d93-4ea3-8bf1-db590b93adfd   20Gi       RWO               balanced-retain-csi   6m5s

I deleted and re-created the Elasticsearch instance with the hope that due to the Retain policy, the new pods (i.e. their PVCs would bind to the existing PVs (and data wouldn't get lost)

However now my pods of the nodeSet are all in pending state with this error

Events:
  Type     Reason             Age                  From                Message
  ----     ------             ----                 ----                -------
  Warning  FailedScheduling   2m37s                default-scheduler   persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" is being deleted
  Normal   NotTriggerScaleUp  2m32s                cluster-autoscaler  pod didn't trigger scale-up: 2 persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" not found
  Warning  FailedScheduling   12s (x7 over 2m37s)  default-scheduler   persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" not found

Why is this happening?

edit: Here are the corresponding PVs

▶ k get pv 
pvc-b258821b-0d93-4ea3-8bf1-db590b93adfd   20Gi       RWO            Retain           Released   elastic/elasticsearch-data--es-multirolenodes1-2          balanced-retain-csi            20m
pvc-ba157213-67cf-4b81-8fe2-6211b771e62c   20Gi       RWO            Retain           Released   elastic/elasticsearch-data--es-multirolenodes1-0          balanced-retain-csi            22m
pvc-e77dbb00-7cad-419f-953e-f3398e3860f4   20Gi       RWO            Retain           Released   elastic/elasticsearch-data--es-multirolenodes1-1          balanced-retain-csi            21m

There is of course no PVC now

▶ k get pvc                       
No resources found in elastic namespace.

The StorageClass under consideration is using the csi driver for the GCP persistent disk, fwiw

Solution

  • with the hope that due to the Retain policy, the new pods (i.e. their PVCs would bind to the existing PVs (and data wouldn't get lost)

    It is explicitly written in the documentation that this is not what happens. the PVs are not available for another PVC after delete of a PVC.

    the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume.