I am scheduling jobs using Cloud Composer running tasks in Kubernetes. I have setup a new node pool in the same GKE as composer and using this to run the Kubernetes tasks. In that node pool I'm using the default service account but binding the account to the same service account as the Composer node pool, using an IAM policy binding, and enabling workload identity.
However I can see from the errors that the kubernetes service account is lacking permissions for some stuff that the composer service account does have access to. Which doesn't make sense since the kubernetes service account has an IAM Policy binding to the composer service account, so they should have exactly same permissions. But for whatever reason this is not true. Any tips on where to look are greatly appreciated...
Enabling the workload identity on the nodepool was the solution. Using terraform the solution looks like this:
resource "google_container_node_pool" "google_container_node_pool_name" {
workload_metadata_config {
mode = "GKE_METADATA"
}