HPA with different namespaces

Question

I have a kubernetes setup with following

sn.  type          service  namespace
1.   statefulset   rabbitmq rabbitmq
2.   deployment    pods     default
3.   hpa           hpa      default

The metrics is exported on `/apis/custom.metrics.k8s.io/v1beta1 as follows

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "services/rabbitmq_queue_messages_ready",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/rabbitmq_queue_messages_ready",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/rabbitmq_queue_messages_ready",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Configuration of hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: demo-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-pod2
  minReplicas: 1
  maxReplicas: 2
  metrics:
  - type: Object
    object:
      metric:
        name: "rabbitmq_queue_messages_ready"
      describedObject:
        apiVersion: custom.metrics.k8s.io/v1/beta1
        kind: Service
        name: rabbitmq
      target:
        type: Value
        value: 50

Whenever I try to deploy the hpa in default namespace I am getting the following error.

 ScalingActive  False   FailedGetObjectMetric  the HPA was unable to compute the replica count: unable to get metric rabbitmq_queue_messages_ready: Service on default rabbitmq/unable to fetch metrics from custom metrics API: the server could not find the metric rabbitmq_queue_messages_ready for services rabbitmq

But when the same hpa is as set do namespace rabbitmq with hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: demo-hpa
  namespace: rabbitmq
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-pod2
  minReplicas: 1
  maxReplicas: 2
  metrics:
  - type: Object
    object:
      metric:
        name: "rabbitmq_queue_messages_ready"
      describedObject:
        apiVersion: custom.metrics.k8s.io/v1/beta1
        kind: Service
        name: rabbitmq
      target:
        type: Value
        value: 50

and deployment.yaml as

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-pod2
  labels:
    app: test-pod2
  namespace: rabbimtq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-pod2
  template:
    metadata:
      labels:
        app: test-pod2
    spec:
      containers:
        - name: test-pod2
          image: golang:1.16
          command: ["sh", "-c", "tail -f /etc/hosts"]

The deployment works perfect.

Is there a way, I can export the metrics for service rabbitmq from rabbitmq namespace to any other namespaces which I can then use for scaling???

Answer 1

HPA is a namespaced resource. It means that it can only scale Deployments which are in the same Namespace as the HPA itself. That's why it is only working when both HPA and Deployment are in the namespace: rabbitmq. You can check it within your cluster by running:

    kubectl api-resources --namespaced=true | grep hpa
    NAME                        SHORTNAMES   APIGROUP                    NAMESPACED   KIND
    horizontalpodautoscalers    hpa          autoscaling                 true         HorizontalPodAutoscaler

The easiest way to make it work would be to simply set the Namespace value of the HPA to the same Namespace of Deployment which you want to scale. For example, if your deployment is set like below:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: rabbimtq

the HPA must also be in the same Namespace:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  namespace: rabbitmq

In other words: Set the metadata.namespace value of your HPA to the metadata.namespace value of your Deployment as the HPA is not able to scale Deployments outside of its Namespace.

Namespaces are like separate "boxes" in your Cluster. Namespaced resources cannot work outside of the Namespace they are into. They can't see resources that are not in their "box".

By doing it that way there would be no need to reconfigure the custom metrics.