Search code examples
apache-beamdataflow

Unit testing a dataflow/apache beam pipline that takes a side input

Using Dataflow 2.1 sdk

        <dependency>
            <groupId>com.google.cloud.dataflow</groupId>
            <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
            <version>[2.1.0, 2.99)</version>
        </dependency>

As per this documentation setSideInputInGlobalWindow should be available. But I only see below two relevant functions: enter image description here

Below is my full code:

DoFnTester<EventItem,EventItem> fnTester = DoFnTester.of(myDoFn);
    TestPipeline pipeline = TestPipeline.create();

TableRow row= new TableRow();
        atr.set("a", 7);
        atr.set("q", new ArrayList<String>() {{ add("video_referrer"); add("from_homepage");}});

PCollectionView<TableRow> sideInput =  (PCollectionView<TableRow>) pipeline.apply(Create.of(Arrays.asList(atr)));
        Iterable<TableRow> sidVal = Arrays.asList(atr);
        fnTester.setSideInput(sideInput, sidVal); //error on this line

        List<Foo> testOutputs = fnTester.processBundle(Arrays.asList(testInput));

I see setSideInput(PCollectionView<T> sideInput, BoundedWindow window, T value) function here. But in this case how do I get BoundedWindow Any ideas what am I missing here, Thanks.

Solution

  • You can try using GlobalWindow.INSTANCE but generally DoFnTester is targeted to be deprecated, and the recommended way is to always use TestPipeline and PAssert.