kube-apiserver: constantly 5 to 10% CPU: Although there is no single request

I installed kind to play around with Kubernetes.

If I use top and sort by CPU usage (key C), then I see that kube-apiserver is constantly consuming 5 to 10% CPU.

Why?

I don't have installed something up to now:

guettli@p15:~$ kubectl get pods --all-namespaces
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-558bd4d5db-ntg7c                     1/1     Running   0          40h
kube-system          coredns-558bd4d5db-sx8w9                     1/1     Running   0          40h
kube-system          etcd-kind-control-plane                      1/1     Running   0          40h
kube-system          kindnet-9zkkg                                1/1     Running   0          40h
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          40h
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          40h
kube-system          kube-proxy-dthwl                             1/1     Running   0          40h
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          40h
local-path-storage   local-path-provisioner-547f784dff-xntql      1/1     Running   0          40h


guettli@p15:~$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  40h
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   40h


guettli@p15:~$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION

kind-control-plane   Ready    control-plane,master   40h   v1.21.1
guettli@p15:~$ kubectl get nodes --all-namespaces
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   40h   v1.21.1

I am curious. Where does the CPU usage come from? How can I investigate this?

Solution

Even in an empty cluster with just one master node, there are at least 5 components that reach out to the API server on a regular basis:

kubelet for the master node
Controller manager
Scheduler
CoreDNS
Kube proxy

This is because API Server acts as the only entry point for all components in Kubernetes to know what the cluster state should be and take action if needed.

If you are interested in the details, you could enable audit logs in the API server and get a very verbose file with all the requests being made. How to do so is not the goal of this answer, but you can start from the apiserver documentation.