Can we trigger Dataflow job(Apache beam) using template of project B from project A in GCP

Question

I am facing one issue and need help for the same. It's related to GCP Dataflow (Apache Beam).

I have a Dataflow template created in project B and everything works if I run it either by using a person's email or service account.

I am looking for a solution to run the Dataflow template of project B from some other project A using a service account defined in the project A (sa-A@PROJECT-A). sa-A@PROJECT-A has already the necessary permissions in project B.

I have already tried using the gcloud command below

gcloud dataflow jobs run BigQueryToBigQuery \
    --gcs-location gs://{GCS bucket}/templates/BigQueryToBigQuery \
    --parameters query=bigQueryTableName={projectID}:{dataset}.{table} \
    --region=us-east1

where the gcs-location used is the location of the template present in the project B.

When I use the service account of project A, it triggers job in project A but not in project B. When I run it using Project B's service account the error Current user cannot act as service account... is raised.

Any help will be appreciated.

Answer 1

I found the solution, just thought to post here so that it will be helpful for other people too.

If we need to trigger a Dataflow job from the project A and to run it on project B then one has to use the above command with the --project flag. The full command would be:

gcloud dataflow jobs run BigQueryToBigQuery \
    --gcs-location gs://{GCS_bucket}/templates/BigQueryToBigQuery \
    --parameters query=bigQueryTableName={projectA_ID}:{dataset}.{table} \
    --region=us-east1 \
    --project=projectB_ID