gcp trigger dataflow job from composer error

Question

I am trying to run Dataflow job from Composer Airflow DAG using below code.

I am getting 2 types of error messages depending on the code.

Please suggest how to fix it.

a) Error 1 : When the Service Account eMail is commented (#)

# "serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",

Error:

Error: Required 'compute.subnetworks.get' permission for 'projects/vpc-host/regions/us-central1/subnetworks/sbn-dataflow'

b) Error 2 : When the Service Account eMail is used

"serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",

Error:

Current user cannot act as service account service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com

Code:

import datetime

from airflow import models
from airflow.contrib.operators.dataflow_operator import DataflowTemplateOperator
from airflow.utils.dates import days_ago

bucket_path = models.Variable.get("bucket_path")
project_id = models.Variable.get("project_id")
gce_zone = models.Variable.get("gce_zone")


default_args = {
    "owner": "Airflow",
    "start_date": days_ago(1),
    "depends_on_past": False,
    "dataflow_default_options": {
        "project": project_id,
        "zone": gce_zone,
        "serviceAccountEmail": "service-7276363xxxxx@cloudcomposer-accounts.iam.gserviceaccount.com",
        "subnetwork": "https://www.googleapis.com/compute/v1/projects/vpc-host/regions/us-central1/subnetworks/sbn-dataflow",
        "tempLocation": bucket_path + "/tmp/",
    }
}


with models.DAG(
    dag_id="composer_dataflow_dag",
    default_args=default_args,
    schedule_interval=datetime.timedelta(days=1)
) as dag:
    dataflow_template_job = DataflowTemplateOperator(
        task_id="dataflow_csv_to_bq",
        template="gs://dataflow-templates/latest/GCS_Text_to_BigQuery",
        parameters={
            "javascriptTextTransformFunctionName": "transformCSVtoJSON",
            "javascriptTextTransformGcsPath": bucket_path + "/SCORE_STG.js",
            "JSONPath": bucket_path + "/SCORE_STG.json",
            "inputFilePattern": bucket_path + "/stg_data.csv",
            "outputTable": project_id + ":gcp_stage.SCORE_STG",
            "bigQueryLoadingTemporaryDirectory": bucket_path + "/tmp/",
        },
        dag=dag,
    )

Answer 1

You have to use different service accounts.Remember that it have to had the access to the resources. That should fix both issues.

You can create a service account to act as a worker as explained on Role Assignment. ie: a worker and a admin.

Besides that I don't find anything outside of the normal. Even the parameters are correctly passed. For other users reference:

• GCS_Text_to_BigQuery
• dataflow_default_options