Search code examples
kuberneteskubectlkubernetes-podamazon-eks

CPU based horizontal pod autoscaling doesn't work in kubernetes cluster

I am trying to test the horizontal pod autoscale feature for my Kubernetes cluster deployed in AWS(using EKS).

I have set Resource as 'CPU' and target type as 'Utilization' and set 'averageUtilization' to 15 in the YAML file.

So technically when the CPU utilization percentage meets the 15% set percentage, Kubernetes cluster should auto-scale horizontally(add pods automatically). This is an important part of the YAML file:

    maxReplicas: 11
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 15

I am pumping a lot of HTTP traffic into the Kubernetes cluster, but all pods stabilize at 35% - 36% CPU with 5 or 6 pods and they do not create additional pods to bring down average CPU utilization to 15% (which I set in YAML file as 'averageUtilization').

No matter how much I debug and run tests many times, it's the same case. What am I doing wrong here, am I missing something else?

Solution

  • Did you check that metrics-server is enabled? It is required in order to enable hpa: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis.

    Here's metrics-server doc: https://github.com/kubernetes-sigs/metrics-server. In order to test it quickly and check it solves your issue, you can disable TLS: --kubelet-insecure-tls. In production, you'll need to enable TLS again.

    To check that metrics-server is enabled, you can run the kubectl top command. It requires a few seconds to start, because it needs to collect some metrics first.