Search code examples
google-cloud-platformgoogle-cloud-dataflowgoogle-cloud-iam

(gcloud.dataflow.flex-template.build) PERMISSION_DENIED: The caller does not have permission

I'm trying to build a flex-template image using a service account:

        gcloud dataflow flex-template build "$TEMPLATE_PATH" \
            --image-gcr-path "$TEMPLATE_IMAGE" \
            --sdk-language "JAVA" \
            --flex-template-base-image JAVA11 \
            --metadata-file "metadata.json" \
            --jar "target/XXX.jar" \
            --env FLEX_TEMPLATE_JAVA_MAIN_CLASS="XXX"

The service account has the following roles:

  "roles/appengine.appAdmin",
  "roles/bigquery.admin",
  "roles/cloudfunctions.admin",
  "roles/cloudtasks.admin",
  "roles/compute.viewer",
  "roles/container.admin",
  "roles/dataproc.admin",
  "roles/iam.securityAdmin",
  "roles/iam.serviceAccountAdmin",
  "roles/iam.serviceAccountUser",
  "roles/iam.roleAdmin",
  "roles/resourcemanager.projectIamAdmin",
  "roles/pubsub.admin",
  "roles/serviceusage.serviceUsageAdmin",
  "roles/servicemanagement.admin",
  "roles/spanner.admin",
  "roles/storage.admin",
  "roles/storage.objectAdmin",
  "roles/firebase.admin",
  "roles/cloudconfig.admin",
  "roles/vpcaccess.admin",
  "roles/compute.instanceAdmin.v1",
  "roles/dataflow.admin",
  "roles/dataflow.serviceAgent"

However, even with the dataflow.admin and dataflow.serviceAgent roles, my service account is still unable to perform this task.

The documentation https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates advises to grant the roles/owner role to the service account, but I'm hesitant to do that as this is meant to be part of a CI/CD pipeline and giving a service account an owner role doesn't really make sense to me unless I'm completely wrong.

Is there any way to circumvent this issue without granting the owner role to the service account?

Solution

  • I just ran into the exact same issue and spent a few hours figuring this out. We use terraform service account as well. As you mentioned there are 2 main issues: service account access and the build logs access.

    1. By default, cloud build will use a default service account of form [project_number]@cloudbuild.gserviceaccount.com so you need to grant permissions to this service account to write to your gcs bucket backing the gcr container registry. I granted roles/storage.admin to my service account.
    2. Like you mentioned, by default again, cloud build saves the logs at gs://[project_number].cloudbuild-logs.googleusercontent.com. This seems to be a hidden bucket in the project, at least I could not see it. In addition, can't configure google_storage_bucket_iam_member for it, instead the recommendation as per this doc is to give roles/viewer at the project level to the service account running the gcloud dataflow ... command.

    I was able to run the command successfully after the above changes.