I am working on a k8s controller using the k8s controller-runtime's setupWithManager approach.
During my reconciliation loop, on mgr.GetClient().Status().Update(...) I sometimes hit Operation cannot be fulfilled...: the object has been modified; please apply your changes to the latest version and try again
During reconciliation, as a first step I obtain my resource and use the same reference throughout the whole loop run.
In my scenario, the version conflict is really unexpected because there isn't any other component interacting with my resource.
The mgr.GetClient() function has the following comment
// GetClient returns a client configured with the Config. This client may
// not be a fully "direct" client -- it may read from a cache, for
// instance. See Options.NewClient for more information on how the default
// implementation works.
GetClient() client.Client
So that got me thinking, is it possible mgr.GetClient().Status().Update(...) writes to the server, however, the cache from which mgt.GetClient().Get(...) reads is not immediately updated by the status update call and thus during next reconciliation, when I make the Get call I read stale data and so experience the version conflict?
I discovered https://github.com/kubernetes-sigs/controller-runtime/issues/245 and https://github.com/kubernetes-sigs/controller-runtime/issues/1959 which confirm my hypothesis
Write path goes to the k8s api-server, however, doesn't update the local cache from which the client reads. The local cache from which the client reads is updated only through a watch event received from the established connection with api-server. Since that can take some time, it is possible for consecutive reconcile to receive stale data on initial resource read