JOB to Dataflow using function

Question

Parameter google function

eu quero passar parameter para query

template file:

"steps":[
      {
         "kind":"ParallelRead",
         "name":"s1",
         "properties":{
            "bigquery_export_format":"FORMAT_AVRO",
            "bigquery_flatten_results":true,
            "bigquery_query":"select * from `myproject2497.teste.teste`",
            "bigquery_use_legacy_sql":false,
            "display_data":[
               {
                  "key":"source",
                  "label":"Read Source",
                  "namespace":"apache_beam.io.iobase.Read",
                  "shortValue":"BigQuerySource",
                  "type":"STRING",
                  "value":"apache_beam.io.gcp.bigquery.BigQuerySource"
               },
               {
                  "key":"query",
                  "label":"Query",
                  "namespace":"apache_beam.io.gcp.bigquery.BigQuerySource",
                  "type":"STRING",
                  "value":"select * from `myproject2497.teste.teste limit 1`"
               },

function google cloud: blob.upload_from_string(timestamp)

parameters = {"bql": bql}
jobname = "poc"    
gcsPath="gs://exemplebucket1321/teste/templates/Bee"
body = {
    "jobName": "{jobname}".format(jobname=jobname),
    "parameters": parameters
}

Answer 1

Please see the documentation on creating:

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates

And RunningTemplates:

https://cloud.google.com/dataflow/docs/guides/templates/running-templates

Note that you will only be pass in parameters in the template code which is already using a ValueProvider (without changing the source). See the link below for the list of IOs with ValueProvider options. For BigQuery you will need Java SDK 2.0+

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#pipeline-io-and-runtime-parameters

Since you are using BigQuery as the source, I believe following the instructiosn in the above two links will resolve your issue