How to get the latest change time of StatefulSet in k8s

Question

I know, for example, that you can get the lastUpdateTime of a Deployment with kubectl:

kubectl get deploy <deployment-name> -o jsonpath={.status.conditions[1].lastUpdateTime}

Or via client-go:

func deploymentCheck(namespace string, clientset *kubernetes.Clientset) bool {
    // get the deployments in the namespace
    deployments, err := clientset.AppsV1().Deployments(namespace).List(context.TODO(), metav1.ListOptions{})
    if errors.IsNotFound(err) {
        log.Fatal("\nNo deployments in the namespace", err)
    } else if err != nil {
        log.Fatal("\nFailed to fetch deployments in the namespace", err)
    }

    var dptNames []string
    for _, dpt := range deployments.Items {
        dptNames = append(dptNames, dpt.Name)
    }
    // check the last update time of the deployments
    for _, dpt := range deployments.Items {
        lastUpdateTime := dpt.Status.Conditions[1].LastUpdateTime
        dptAge := time.Since(lastUpdateTime.Time)
        fmt.Printf("\nDeployment %v age: %v", dpt.Name, dptAge)
    }
}

The equivalent of lastUpdateTime := dpt.Status.Conditions[1].LastUpdateTime for a StatefulSet doesn't seem to exist.

So, how can I get the lastUpdateTime of a StatefulSet?

Answer 1

I noticed that the only things that change after someone edits a given resource are the resource's lastAppliedConfiguration, Generation and ObservedGeneration. So, I stored them in lists:

    for _, deployment := range deployments.Items {
        deploymentNames = append(deploymentNames, deployment.Name)
        lastAppliedConfig := deployment.GetAnnotations()["kubectl.kubernetes.io/last-applied-configuration"]

        lastAppliedConfigs = append(lastAppliedConfigs, lastAppliedConfig)
        generations = append(generations, deployment.Generation)
        observedGenerations = append(observedGenerations, deployment.Status.ObservedGeneration)
    }

Here's the full function:

func DeploymentCheck(namespace string, clientset *kubernetes.Clientset) ([]string, []string, []int64, []int64) {

    var deploymentNames []string
    var lastAppliedConfigs []string
    var generations []int64
    var observedGenerations []int64

    deployments, err := clientset.AppsV1().Deployments(namespace).List(context.TODO(), metav1.ListOptions{})
    if errors.IsNotFound(err) {
        log.Print("No deployments in the namespace", err)
    } else if err != nil {
        log.Print("Failed to fetch deployments in the namespace", err)
    }

    for _, deployment := range deployments.Items {
        deploymentNames = append(deploymentNames, deployment.Name)
        lastAppliedConfig := deployment.GetAnnotations()["kubectl.kubernetes.io/last-applied-configuration"]

        lastAppliedConfigs = append(lastAppliedConfigs, lastAppliedConfig)
        generations = append(generations, deployment.Generation)
        observedGenerations = append(observedGenerations, deployment.Status.ObservedGeneration)
    }

    return deploymentNames, lastAppliedConfigs, generations, observedGenerations
}

I use all this information to instantiate a struct called Namespace, which contains all major resources a k8s namespace can have.

Then, after a given time I check the same namespace again and check if its resources had any changes:

if !reflect.DeepEqual(namespace.stsLastAppliedConfig, namespaceCopy.stsLastAppliedConfig) {
...
}
else if !reflect.DeepEqual(namespace.stsGeneration, namespaceCopy.stsGeneration) {
...
}
else if !reflect.DeepEqual(namespace.stsObservedGeneration, namespaceCopy.stsObservedGeneration) {
...
}

So, the only workaround I found was to compare the resource's configuration, including StatefulSets', after a given time. Apparently, for some resources you cannot get any information about their lastUpdateTime.

I also found out that lastUpdateTime is actually not reliable, as it understands minor cluster changes as the resource's change. For example, if a cluster rotates and kills all pods, the lastUpdateTime of a Deployment will update its time. That's not what I wanted. I wanted to detect user changes to resources, like when someone applies an edited yaml file or run kubectl edit.

@hypperster , I hope it helps.