I am using the following command to gracefully delete any stale pods in Pending state:
kubectl get pod -n my-namespace | grep Pending | awk '{print $1}' | xargs kubectl delete pod -n my-namespace
However, I would like to add a condition that deletes only those pods who have been in pending state for more than N hours. There is the AGE column returned with get pods but its time unit varies and I am assuming there is a better way.
Also would appreciate if anyone can mention any best practices around this as I aim to run this command periodically to cleanup the pending Pods.
It is difficult to calculate how much time a Pod has spent in a particular Status by using kubectl solely and without a help of some 3rd party tools. However, I got a solution that you may find useful.
You can list all Pods that are in Pending state and are older than X days. For example, the command below will list all Pending Pods that are older than 5 days:
kubectl get pods --field-selector=status.phase=Pending --sort-by=.metadata.creationTimestamp | awk 'match($5,/[6-9]d|[0-9][0-9]d|[0-9][0-9][0-9]d/) {print $0}'
Than you can use the next command to delete these pods:
kubectl delete pod $(kubectl get pods --field-selector=status.phase=Pending --sort-by=.metadata.creationTimestamp | awk 'match($5,/[6-9]d|[0-9][0-9]d|[0-9][0-9][0-9]d/) {print $0}')
The value can and should be adjusted by modifying the awk scripting in order to match your use case.