I am currently using Cloud Build for my Dataflow Flex template to kick off jobs. Here's my current command:
gcloud beta builds submit --config run.yaml --substitutions _REGION=$REGION \
--substitutions _FMPKEY=$FMPKEY --no-source
Currently this is running fine from Cloud Shell. But now I want the build to be kicked off based on a trigger.. So I created a Cloud Build that will trigger running this file based on dropping a message to a topic:
https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/pipeline/run.yaml
However, after publishing a message to the selected topic, all my builds fail with the following error:
our build failed to run: generic::invalid_argument:generic::invalid_argument:
'build.service_account' field cannot be set for triggered builds
I cannot see any logs or details, so it's not clear to me what is going on.. I am guessing it has something to do with the last line in my run.yaml?
options:
logging: CLOUD_LOGGING_ONLY
# Use the Compute Engine default service account to launch the job.
serviceAccount: projects/$PROJECT_ID/serviceAccounts/$PROJECT_NUMBER-compute@developer.gserviceaccount.com
However I see no option for selecting the service account in cloud build. Do I need to set some permissions in IAM?
You are correct with your guess and this is working as intended.
Cloud Build has a default service account to execute builds on your behalf. While GCP allows you to configure user-specific accounts for additional control, it doesn't apply when you're using build triggers. Build triggers only use the default service account to execute builds.
This is documented in GCP docs:
Build triggers use Cloud Build service account to execute builds. This could provide elevated build-time permissions to users who use triggers to start a build. Keep the following security implications in mind when using build triggers ...
Also as part of limitation:
User-specified service accounts only work with manual builds; they don't work with build triggers.
Therefore, you must pass a config yaml without serviceAccount
if you plan on using build triggers.