I am running load tests using K6 as a tool. I am sending 1000 requests per second to a .NET API (without database) that I deployed in Kubernetes. Sending this many requests impacts the performance of my application. This is noticeable in the request duration(as expected).
The K8s cluster has two nodes and one of them has high CPU usage because of all the requests I am sending (which is also expected). However, the second node has low CPU usage. While monitoring the CPU usage of the pods I am seeing the same results. Both pods use about 10% - 20% of their CPU. How can these metrics be explained? Shouldn't the loadbalancer (ingress) balance the load evenly? And why am I seeing a decrease in performance if I still have resoures left?
Any advice/help would be greatly appreciated.
Node 1, CPU usage: https://i.sstatic.net/r0QPF.png
Node 2, CPU usage: https://i.sstatic.net/Ovub5.png
Pod 1, CPU usage: https://i.sstatic.net/ofUmm.png
Pod 2, CPU usage: https://i.sstatic.net/Jj3sC.png
Did you make sure that the pods are scheduled one on each node? Otherwise, they could be both scheduled on the same node. For your test with one per node, a daemonset would be sufficient. Otherwise, consider topology spread constraints:
https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/