Search code examples
kubernetescontainersjobsrbac

How to run kubectl within a job in a namespace?


Hi I saw this documentation where kubectl can run inside a pod in the default pod. Is it possible to run kubectl inside a Job resource in a specified namespace? Did not see any documentation or examples for the same..

When I tried adding serviceAccounts to the container i got the error:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:my-namespace:internal-kubectl" cannot list resource "pods" in API group "" in the namespace "my-namespace"

This was when i tried sshing into the container and running the kubctl.

Editing the question.....

As I mentioned earlier, based on the documentation I had added the service Accounts, Below is the yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-kubectl  
  namespace: my-namespace   
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: modify-pods
  namespace: my-namespace
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
      - delete      
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: modify-pods-to-sa
  namespace: my-namespace
subjects:
  - kind: ServiceAccount
    name: internal-kubectl
roleRef:
  kind: Role
  name: modify-pods
  apiGroup: rbac.authorization.k8s.io      
---
apiVersion: batch/v1
kind: Job
metadata:
  name: testing-stuff
  namespace: my-namespace
spec:
  template:
    metadata:
      name: testing-stuff
    spec:
      serviceAccountName: internal-kubectl
      containers:
      - name: tester
        image: bitnami/kubectl
        command:
         - "bin/bash"
         - "-c"
         - "kubectl get pods"
      restartPolicy: Never 

On running the job, I get the error:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:my-namespace:internal-kubectl" cannot list resource "pods" in API group "" in the namespace "my-namespace"

Solution

  • Is it possible to run kubectl inside a Job resource in a specified namespace? Did not see any documentation or examples for the same..

    A Job creates one or more Pods and ensures that a specified number of them successfully terminate. It means the permission aspect is the same as in a normal pod, meaning that yes, it is possible to run kubectl inside a job resource.

    TL;DR:

    • Your yaml file is correct, maybe there were something else in your cluster, I recommend deleting and recreating these resources and try again.
    • Also check the version of your Kubernetes installation and job image's kubectl version, if they are more than 1 minor-version apart, you may have unexpected incompatibilities

    Security Considerations:

    • Your job role's scope is the best practice according to documentation (specific role, to specific user on specific namespace).
    • If you use a ClusterRoleBinding with the cluster-admin role it will work, but it's over permissioned, and not recommended since it's giving full admin control over the entire cluster.

    Test Environment:

    • I deployed your config on a kubernetes 1.17.3 and run the job with bitnami/kubectl and bitnami/kubectl:1:17.3. It worked on both cases.
    • In order to avoid incompatibility, use the kubectl with matching version with your server.

    Reproduction:

    $ cat job-kubectl.yaml 
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: testing-stuff
      namespace: my-namespace
    spec:
      template:
        metadata:
          name: testing-stuff
        spec:
          serviceAccountName: internal-kubectl
          containers:
          - name: tester
            image: bitnami/kubectl:1.17.3
            command:
             - "bin/bash"
             - "-c"
             - "kubectl get pods -n my-namespace"
          restartPolicy: Never 
    
    $ cat job-svc-account.yaml 
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: internal-kubectl  
      namespace: my-namespace   
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: modify-pods
      namespace: my-namespace
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "list", "delete"]      
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: modify-pods-to-sa
      namespace: my-namespace
    subjects:
      - kind: ServiceAccount
        name: internal-kubectl
    roleRef:
      kind: Role
      name: modify-pods
      apiGroup: rbac.authorization.k8s.io
    
    • I created two pods just to add output to the log of get pods.
    $ kubectl run curl --image=radial/busyboxplus:curl -i --tty --namespace my-namespace
    the pod is running
    $ kubectl run ubuntu --generator=run-pod/v1 --image=ubuntu -n my-namespace
    pod/ubuntu created
    
    • Then I apply the job, ServiceAccount, Role and RoleBinding
    $ kubectl get pods -n my-namespace
    NAME                    READY   STATUS      RESTARTS   AGE
    curl-69c656fd45-l5x2s   1/1     Running     1          88s
    testing-stuff-ddpvf     0/1     Completed   0          13s
    ubuntu                  0/1     Completed   3          63s
    
    • Now let's check the testing-stuff pod log to see if it logged the command output:
    $ kubectl logs testing-stuff-ddpvf -n my-namespace
    NAME                    READY   STATUS    RESTARTS   AGE
    curl-69c656fd45-l5x2s   1/1     Running   1          76s
    testing-stuff-ddpvf     1/1     Running   0          1s
    ubuntu                  1/1     Running   3          51s
    

    As you can see, it has succeeded running the job with the custom ServiceAccount.

    Let me know if you have further questions about this case.