Search code examples
google-cloud-dataflow

How to disable public ip in a predefined template for a dataflow job launch

I am trying to deploy a dataflow job using google's predefined template using python api

I do not want my dataflow compute instance to have a public ip, so I use something like this:

GCSPATH="gs://dataflow-templates/latest/Cloud_PubSub_to_GCS_Text"
BODY = {
    "jobName": "{jobname}".format(jobname=JOBNAME),
    "parameters": {
        "inputTopic" : "projects/{project}/topics/{topic}".format(project=PROJECT, topic=TOPIC),
        "outputDirectory": "gs://{bucket}/pubsub-backup-v2/{topic}/".format(bucket=BUCKET, topic=TOPIC),
        "outputFilenamePrefix": "{topic}-".format(topic=TOPIC),
        "outputFilenameSuffix": ".txt"
     },
     "environment": {
        "machineType": "n1-standard-1",
        "usePublicIps": False,
        "subnetwork": SUBNETWORK,
     }
}

request = service.projects().templates().launch(projectId=PROJECT, gcsPath=GCSPATH, body=BODY)
response = request.execute()

but I get this error:

raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://dataflow.googleapis.com/v1b3/projects/ABC/templates:launch?alt=json&gcsPath=gs%3A%2F%2Fdataflow-templates%2Flatest%2FCloud_PubSub_to_GCS_Text returned "Invalid JSON payload received. Unknown name "use_public_ips" at 'launch_parameters.environment': Cannot find field.">

If I remove the usePublicIps, it goes through, but my compute instance gets deployed with public ip.

Solution

  • I found one way to make this work

    1. Clone Google Defined Templates

    2. Run the template with custom parameters

    mvn compile exec:java \
 -Dexec.mainClass=com.google.cloud.teleport.templates.PubsubToText \
 -Dexec.cleanupDaemonThreads=false \
 -Dexec.args=" \
 --project=${PROJECT_ID} \
 --stagingLocation=gs://${BUCKET}/dataflow/pipelines/${PIPELINE_FOLDER}/staging \
 --tempLocation=gs://${BUCKET}/dataflow/pipelines/${PIPELINE_FOLDER}/temp \
 --runner=DataflowRunner \
 --windowDuration=2m \
 --numShards=1 \
 --inputTopic=projects/${PROJECT_ID}/topics/$TOPIC \
 --outputDirectory=gs://${BUCKET}/temp/ \
 --outputFilenamePrefix=windowed-file \
 --outputFilenameSuffix=.txt \
 --workerMachineType=n1-standard-1 \
 --subnetwork=${SUBNET} \
 --usePublicIps=false"