Invoke-AzAksRunCommand how to run aks-command pod with toleration

Question

I'm trying to run:

Invoke-AzAksRunCommand -ResourceGroupName myrg -Name mycluster -Command "kubectl delete pod --all -n myns;"

but the temporary pod that Invoke-AzAksRunCommand will create to execute the command inside, stay stuck becouse on my cluster there are some toleration. Is it possbile to specify the toleration in some way? On microsoft doc i didn't see nothing.

Thanks

Answer 1

The Invoke-AzAksRunCommand command doesn't support directly specifying tolerations for the temporary pod it creates. This is a limitation in the way the command works, as it's designed for quick execution of kubectl commands inside the AKS cluster without much customization of the pod that runs the command.

As a workaround, you could manually run a pod with tolerations and use a job or pod to execute the command. Example using tolerations:

apiVersion: v1
kind: Pod
metadata:
  name: temp-command-runner
spec:
  containers:
  - name: kubectl
    image: bitnami/kubectl
    command: ["kubectl", "delete", "pod", "--all", "-n", "myns"]
  tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"
  restartPolicy: Never

note that you need to grant the default service account in the myns namespace the appropriate permissions using a Role and RoleBinding.

Now when you do

kubectl get pods -n myns

the command (kubectl delete pod --all -n myns) inside the pod was successfully executed, and all the pods in the myns namespace were deleted, including the temp-command-runner itself. Since the pod had restartPolicy: Never, it did not restart and was removed after completing its task.

Second option is using a Kubernetes Job with tolerations. You can just modify the same YAML to use a Kubernetes Job resource.

apiVersion: batch/v1
kind: Job
metadata:
  name: temp-command-runner
  namespace: myns
spec:
  template:
    spec:
      containers:
      - name: kubectl
        image: bitnami/kubectl
        command: ["/bin/sh", "-c", "kubectl delete pod --all -n myns"]
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "custom-key"
        operator: "Equal"
        value: "custom-value"
        effect: "NoSchedule"
      restartPolicy: Never
  backoffLimit: 1

Checkout-

K8s taints and toleration doc

tutorial using taints and toleration