Search code examples
kubernetesetcdkubernetes-apiserverkube-apiserver

kube-apiserver: force etcd worker key to be re-constructed from api-server


Problem: I am creating for testing new node conditions and status, when i am removing them from my code or controller they remain in etcd /kubernetes.io/minions/. whenever i rune describe.

I can't update it as it's encrypted, I want a way to remove unwanted status, node condition from the etcd or force all status to be deleted and re-constructed from apiserver.

Tried to delete the minion key and reboot the worker but this is not a practical fix, any ideas or workaround ?


Solution

  • I have found that one way to do so is to delete the node key from etcd and restart kubelet in the node. This re-construct the key back again.

    Workaround: etcdctl get /kubernetes.io/minions/

    Keep in mind that api sends to it binary proto, so i used "grep -a" or ad to etcdctl get -w json to decode any binary.

    Fix: etcdctl del /kubernetes.io/minions/ Restart kubelet in the worker node minion key in etcd will be re-constructed

    Fix2: Delete all condition by curl the kube api

    curl -k -H "Authorization: Bearer $TOKEN -H "Content-Type: application/json-patch+json" -X PATCH https://APISERVER:6443/api/v1/nodes/f<Node-Name>/status --data '[{ "op": "remove", "path": "/status/conditions"}]'