Search code examples
kuberneteskubernetes-rback8s-serviceaccount

Cannot list or delete ClusterRole or ClusterRoleBinding with a Kubernetes ServiceAccount


I want to create a Kubernetes CronJob that deletes resources (Namespace, ClusterRole, ClusterRoleBinding) that may be left over (initially, the criteria will be "has label=Something" and "is older than 30 minutes". (Each namespace contains resources for a test run).

I created the CronJob, a ServiceAccount, a ClusterRole, a ClusterRoleBinding, and assigned the service account to the pod of the cronjob.

The cronjob uses an image that contains kubectl, and some script to select the correct resources.

My first draft looks like this:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  concurrencyPolicy: Forbid
  schedule: "*/1 * * * *"
  jobTemplate:
    # job spec
    spec:
      template:
        # pod spec
        spec:
          serviceAccountName: my-app
          restartPolicy: Never
          containers:
          - name: my-app
            image: image-with-kubectl
            env:
            - name: MINIMUM_AGE_MINUTES
              value: '2'
            command: [sh, -c]
            args:
            # final script is more complex than this
            - |
              kubectl get namespaces
              kubectl get clusterroles
              kubectl get clusterrolebindings
              kubectl delete Namespace,ClusterRole,ClusterRoleBinding --all-namespaces --selector=bla=true

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-app
  labels:
    app: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-app
subjects:
  - kind: ServiceAccount
    name: my-app
    namespace: default
    apiGroup: ""

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-app
  labels:
    app: my-app
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - clusterroles
      - clusterrolebindings
    verbs: [list, delete]

The cronjob is able to list and delete namespaces, but not cluster roles or cluster role bindings. What am I missing?

(Actually, I'm testing this with a Job first, before moving to a CronJob):

NAME              STATUS   AGE
cattle-system     Active   16d
default           Active   16d
fleet-system      Active   16d
gitlab-runner     Active   7d6h
ingress-nginx     Active   16d
kube-node-lease   Active   16d
kube-public       Active   16d
kube-system       Active   16d
security-scan     Active   16d
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:default:my-app" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:default:my-app" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:default:my-app" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:default:my-app" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope`

Solution

  • You need to change your ClusterRole like this :

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: my-app
      labels:
        app: my-app
    rules:
      - apiGroups: [""]
        resources:
          - namespaces
        verbs: [list, delete]
      - apiGroups: ["rbac.authorization.k8s.io"]
        resources:
          - clusterroles
          - clusterrolebindings
        verbs: [list, delete]
    

    The ressources are now in the right apiGroup