We're trying to build some simple automation to detect if a deployment has finished a rolling update or not, or if a rolling update has failed. The naive way (which we do today) is to simply get all the pods for the deployment, and wait until they're ready. We look at all the pods, and if one of the pods have restarted 3 or more times (since we started the update), we roll back the update. This works fine most of the time. The times it doesn't, is when the current version that is deployed is in a failing state for any reason, so that the existing pods are continuously restarting, triggering a rollback from our end.
So the idea I had was to monitor the pods in the new replicaset that is being rolled out after I initiate a rolling update. This way we won't detect failing pods in the previous version as failures of the rolling update. I have found out how to find the pods within a replicaset (PS: I use powershell as my shell but hopefully the translation to bash or whatever you prefer is fairly straight forward):
kubectl get pod -n <namespace> -l <selector> -o json | ConvertFrom-Json | Where {$_.metadata.ownerReferences.name -eq <replicaset-name>}
And I can easily find which replicaset belongs to a deployment using the same method, just querying for replicasets and filtering on their ownerReference again.
HOWEVER, a deployment has at least two replicasets during a rolling update: The new, and the old. I can see the names of those if I use "kubectl describe deployment" -- but that's not very automation-friendly without some text processing. It seems like a fragile way to do it. It would be great if I could find the same information in the json, but seems that doesn't exist.
So, my question is: How do I find the connection between the deployments and the current/old replicaset, preferably in JSON format? Is there some other resource that "connects" these two, or is there some information on the replicasets themselves I can use to distinguish the new from the old replicaset?
The deployment will indicate the current "revision" of the replica set with the deployment.kubernetes.io/revision annotation.
metadata:
annotations:
deployment.kubernetes.io/revision: "4"
This will exist on both the deployment and the replicaset. The replicaset with revision N-1 will be the "old" one.