Resolved deployment dataflow with cloud deployment manager

Question

I registered dataflow with command : gcloud deployment-manager type-providers create dataflow --descriptor-url='https://dataflow.googleapis.com/$discovery/rest?version=v1b3'

When i run this script

- name: "my-topic-to-avro"
  type: 'project_id/dataflow:dataflow.projects.locations.templates.launch'
  properties:
    projectId: project_id
    gcsPath: "gs://test-create-123"
    jobName: "my-topic-to-avro"
    location: "europe-west1"
    parameters:
      inputTopic: "projects/project_id/topics/test"
      outputDirectory: "gs://test-create-123/"
      avroTempDirectory: "gs://test-create-123/"

In output i have this :

ERROR: (gcloud.beta.deployment-manager.deployments.update) Error in Operation [operation-1598980583c2a0ec69]: errors:
- code: RESOURCE_ERROR
  location: /deployments/quick-deployment/resources/my-topic-to-avro
  message: '{"ResourceType":"project_id/dataflow:dataflow.projects.locations.templates.launch","ResourceErrorCode":"401","ResourceErrorMessage":{"code":401,"message":"Request
    is missing required authentication credential. Expected OAuth 2 access token,
    login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.","status":"UNAUTHENTICATED","statusMessage":"Unauthorized","requestPath":"https://dataflow.googleapis.com/v1b3/projects/project_id/locations/europe-west1/templates:launch","httpMethod":"POST"}}'

I can have my token run command : gcloud auth print-access-token, but i don't know where insert my value and the schema for my yaml to insert all value to create dataflow.

Any help appreciated.

Answer 1

The "401 - Request is missing required authentication credential" error message that is triggered when doing a POST request to the Dataflow API is due to a missing credential. The following public reference explains in detail how to use OAuth 2.0 to access Google APIs. Please read it carefully and make sure to follow the steps as mentioned to avoid any errors.

Another place to check this is double check that the Dataflow API is enabled and try setting "GOOGLE_APPLICATION_CREDENTIALS" to point to the JSON of the service account you are using.

I found this Authentication documentation where is mentioned that you can use the Authorization header to supply an access token from the project's service account, you can also try with this method but is important t o clarify that is an example for GKE cluster.

I noticed this question was addressed in this post.