Setting up a Kubernetes Cronjob: Cannot get resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "staging"

Question

I've tried to setup a cron job to patch a horizontal pod autoscaler, but the job returns horizontalpodautoscalers.autoscaling "my-web-service" is forbidden: User "system:serviceaccount:staging:sa-cron-runner" cannot get resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "staging"

I've tried setting up all the role permissions as below:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: staging
  name: cron-runner
rules:
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["patch"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cron-runner
  namespace: staging
subjects:
- kind: ServiceAccount
  name: sa-cron-runner
  namespace: staging
roleRef:
  kind: Role
  name: cron-runner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cron-runner
  namespace: staging
---

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: scale-up-job
  namespace: staging
spec:
  schedule: "16 20 * * 1-6"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: scale-up-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa my-web-service --patch '{\"spec\":{\"minReplicas\":6}}'
          restartPolicy: OnFailure

kubectl auth can-i patch horizontalpodautoscalers -n staging --as sa-cron-runner also returns no, so the permissions can't be setup correctly.

How can I debug this? I can't see how this is setup incorrectly

Answer 1

I think that the problem is that the cronjob needs to first get the resource and then Patch the same. So, you need to explicitly specify the permission to get in the Role spec.

The error also mentions authentication problem with getting the resource:

horizontalpodautoscalers.autoscaling "my-web-service" is forbidden: User "system:serviceaccount:staging:sa-cron-runner" cannot get resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "staging"

Try modifying the verbs part of the Role as:

verbs: ["patch", "get"]

You could also include list and watch depending on the requirements in the scripts that is run inside the cronjob.