Kubernetes too old resource version

I'm working on a operator which create watch for different k8s resources. Every now and then I can see below exception in the logs and application just stop. What is causing this issue and how can I fix this ?

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 29309228 (33284573)
    at [email protected]/io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:263)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Solution

I'm from Fabric8 Kubernetes Client team. I think it's standard behavior of Kubernetes to give 410 after some time during watch. It's usually client's responsibility to handle it. In the context of a watch, it will return HTTP_GONE when you ask to see changes for a resourceVersion that is too old - i.e. when it can no longer tell you what has changed since that version, since too many things have changed. In that case, you'll need to start again, by not specifying a resourceVersion in which case the watch will send you the current state of the thing you are watching and then send updates from that point.

Fabric8 does not handle it with plain watch. But it is handling it in SharedInformer API, see ReflectorWatcher. I would recommend using informer API when writing operators since it's better than plain list and watch. Here is a simple example of Using SharedInformer API:

try (KubernetesClient client = new DefaultKubernetesClient()) {
  SharedInformerFactory sharedInformerFactory = client.informers();
  SharedIndexInformer<Pod> podInformer = sharedInformerFactory.sharedIndexInformerFor(Pod.class, PodList.class, 30 * 1000L);
  podInformer.addEventHandler(new ResourceEventHandler<Pod>() {
    @Override
    public void onAdd(Pod pod) {
      // Handle Creation
    }

    @Override
    public void onUpdate(Pod oldPod, Pod newPod) {
      // Handle update
    }

    @Override
    public void onDelete(Pod pod, boolean deletedFinalStateUnknown) {
      // Handle deletion
    }
  });
  sharedInformerFactory.startAllRegisteredInformers();
}

You can find a full demo of a simple operator using Fabric8 SharedInformer API here: PodSet Operator In Java