Google Cloud DataFlow Deployment Failed

Question

I am trying to deploy GCP flex-template DataFlow via terraform(Resource Creation). I did below steps:

1. Created Docker image and pushed to container registry. Docker file has below code:

ENV PYTHONUNBUFFERED True

ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

# Configure access to GCP
ENV GOOGLE_APPLICATION_CREDENTIALS=creds_key.json
RUN gcloud auth activate-service-account --key-file=creds_key.json


RUN pip3 install --upgrade pip
RUN pip install poetry==1.2.1
RUN poetry self add "keyrings.google-artifactregistry-auth=^1" -v
RUN poetry config virtualenvs.create false
RUN poetry install -v
#RUN rm -rf cred_key.json
# Entry Point for Dataflow Job:
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="./run_pipeline.py"
ENV FLEX_TEMPLATE_PYTHON_SETUP_FILE="./setup.py"

Service Account used to deploy the resource has access to below roles, and above key file is also same.

dataflow worker
storage admin
storage object admin
storage object creator
Service Account User/Actor
Pub/Sub Admin/Subscriber

Error:

Response: {"error":"invalid_scope","error_description":"Invalid OAuth scope or ID token audience provided."}

I got the different error when i removed auth activate-service-account:

I can see that the file creds_key.json is available in image.

Answer 1

This issue can be solved by including name of service account with the command:

gcloud auth activate-service-account ACCOUNT-NAME --key-file=KEY_FILE

For more information you can refer to this document.