Search code examples
kuberneteskubernetes-statefulset

Recreate Pod managed by a StatefulSet with a fresh PersistentVolume

On an occasional basis I need to perform a rolling replace of all Pods in my StatefulSet such that all PVs are also recreated from scratch. The reason to do so is to get rid of all underlying hard drives that use old versions of encryption key. This operation should not be confused with regular rolling upgrades, for which I still want volumes to survive Pod terminations. The best routine I figured so far to do that is following:

  1. Delete the PV.
  2. Delete the PVC.
  3. Delete the Pod.
  4. Wait until all deletions complete.
  5. Manually recreate the PVC deleted in step 2.
  6. Wait for the new Pod to finish streaming data from other Pods in the StatefulSet.
  7. Repeat from step 1. for the next Pod.

I'm not happy about step 5. I wish StatefulSet recreated the PVC for me, but unfortunately it does not. I have to do it myself, otherwise Pod creation fails with following error: 

Warning  FailedScheduling   3s (x15 over 15m)  default-scheduler   persistentvolumeclaim "foo-bar-0" not found

Is there a better way to do that?

Solution

  • This is described in https://github.com/kubernetes/kubernetes/issues/89910. The workaround proposed there, of deleting the new Pod which is stuck pending, works and the second time it gets replaced a new PVC is created. It was marked as a duplicate of https://github.com/kubernetes/kubernetes/issues/74374, and reported as potentially fixed in 1.20.