Search code examples
kubernetesprometheushorizontal-pod-autoscaling

Kubernetes Horizontal Pod Autoscaler fails with unexpected GroupVersion


I am trying to create a HPA using Prometeus and Prometeus Adapter.

I am getting a "unexpected GroupVersion" error in the Status and I just have no idea what it means.

$ kubectl describe hpa
Name:                                             pathology-uploader-hpa
Namespace:                                        default
Labels:                                           <none>
Annotations:                                      <none>
CreationTimestamp:                                Tue, 08 Mar 2022 15:50:12 +0000
Reference:                                        Deployment/pathology-uploader-dep
Metrics:                                          ( current / target )
  "pathology_uploader_avg_process_time" on pods:  <unknown> / 180k
Min replicas:                                     1
Max replicas:                                     100
Deployment pods:                                  0 current / 0 desired
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: unexpected GroupVersion string: /apps/v1
Events:
  Type     Reason          Age                   From                       Message
  ----     ------          ----                  ----                       -------
  Warning  FailedGetScale  4m34s (x41 over 14m)  horizontal-pod-autoscaler  unexpected GroupVersion string: /apps/v1

Here is my spec:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: pathology-uploader-hpa
spec:
  maxReplicas: 100 # define max replica count
  minReplicas: 1   # define min replica count
  scaleTargetRef:
    kind: Deployment
    name: pathology-uploader-dep
    apiVersion: /apps/v1
  metrics:
    - type: Pods
      pods:
        metric:
          name: pathology_uploader_avg_process_time
        target:
          type: AverageValue
          averageValue: 180000  # allow one job to take 3 minutes

I am using the helm charts supplied by the prometheus-community. The rules section that I have supplied to the prometheus-adapter is

rules:
  default: true
  custom:
  - seriesQuery: '{__name__=~"^pathology_uploader_process_time$"}'
    resources:
      template: <<.Resource>>
    name:
      matches: ".*"
      as: "pathology_uploader_avg_process_time"
    metricsQuery: <<.Series>>
prometheus:
  url: http://prometheus-server.default.svc
  port: 80
  path: ""

Solution

  • Update your spec like below.

    "/apps/v1" to "apps/v1"

    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    metadata:
      name: pathology-uploader-hpa
    spec:
      maxReplicas: 100 # define max replica count
      minReplicas: 1   # define min replica count
      scaleTargetRef:
        kind: Deployment
        name: pathology-uploader-dep
        ## update "/apps/v1" to "apps/v1"
        apiVersion: apps/v1