Search code examples
javagoogle-cloud-dataflowapache-beam

How to mentioned Required and Optional Parameters in GCP Dataflow Template?


I have created a custom Dataflow Template that uses a few mandatory and optional parameters. I am able to create a Template but when I open it in Dataflow UI, it does not show my declared required parameters.

I got one annotation @Validation.Required but that is also not working. Is there anything I am missing?

My Options Class is

package org.example;

import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.Validation;
import org.apache.beam.sdk.options.ValueProvider;

public interface TestOptions extends GcpOptions {
    @Description("Project Id")
    @Default.String("Project Id")
    ValueProvider<String> getProjectId();
    void setProjectId(ValueProvider<String> value);

    @Validation.Required
    @Description("Path to the input file")
    @Default.String("gs://nikhil-test-bucket/Test/input.txt")
    ValueProvider<String> getInputFile();
    void setInputFile(ValueProvider<String> value);

    @Validation.Required
    @Description("Path to the output file")
    @Default.String("gs://nikhil-test-bucket/Test/output.txt")
    ValueProvider<String> getOutputFile();
    void setOutputFile(ValueProvider<String> value);

    @Description("Path to the Config file")
    @Default.String("gs://nikhil-test-bucket/Test/config.txt")
    ValueProvider<String> getConfigFile();
    void setConfigFile(ValueProvider<String> value);
}

Solution

  • The Dataflow UI reads the _metadata file (also called spec) for the list of parameters, not the actual source code.

    In the metadata parameter, you can specify "isOptional": true for any parameter.