GCP - Scale GKE pods based on custom logging metric using HPA

Question

I want to use the Custom log metrics on GKE HPA. Metrics are able to view on metrics explorer but unable to use it on HPA . We have installed Custom metrics adapter and We are able to use other custom metrics like kubernetes.io|pod|network|received_bytes_count successfully for scaling. Below image shows the Metrics explorer graph for custom metric that i want to use on HPA

This metric was created from application logs

Used following HPA yaml to use that metric

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name:   "similar-products-rts-hpa"
  namespace: relevancy
spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 120
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: similar-products
  minReplicas: 3
  maxReplicas: 6
  metrics:
    - type: Pods
      pods:
        metric:
          name: "logging.googleapis.com|user|Similar_Products(RTS)_Inbound_Request_Count"
        target:
          type: AverageValue
          averageValue: 25

Please find the error below

The HPA was unable to compute the replica count: unable to get metric logging.googleapis.com|user|Similar_Products(RTS)_Inbound_Request_Count: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest

Answer 1

Unfortunately, upper case letters in metric names are not supported as HPA treats metrics as pseudo resources which means they are not case sensitive. I also believe that parentheses are invalid characters as well for metric names.

Any chance you can change your metric name to lowercase and remove the parentheses? Maybe something like similar_products_rts_inbound_request_count?

EDIT: The other issue I just noticed is that the metric is a container metric and not a pod metric. Prior to this change, it was necessary to modify the custom metrics adapter deployment to support container metrics. You can either update your deployment with the current manifest,or you can modify your current deployment by adding --fallback-for-container-metrics=true:

spec:
      serviceAccountName: custom-metrics-stackdriver-adapter
      containers:
      - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.0-gke.0
        imagePullPolicy: Always
        name: pod-custom-metrics-stackdriver-adapter
        command:
        - /adapter
        - --use-new-resource-model=true
        - --fallback-for-container-metrics=true
        resources:
          limits:
            cpu: 250m
            memory: 200Mi
          requests:
            cpu: 250m
            memory: 200Mi