Search code examples
google-cloud-platformapache-beamsubscriptiongoogle-cloud-pubsubdataflow

Pub/Sub input in templated Dataflow is completely ignoring NestedValueProvider

I've create template for Dataflow which is reading messages from Pub/Sub The problem is that NestedValueProvider is not possible to modify sended via options value and I do not have clue why - below some screens

Passed parameter:

enter image description here

Code:

enter image description here

Result:

enter image description here

If anyone would ask why I needto nest it - I wanted to only require to add subscriptionID and just add prefix with rest of the path..

EDIT

My steps:

1.Created pipeline which is reading from Pub/Sub subscription and for subscription name is using NestedValueProvider for creating subscription name

pipeline.apply(
                            "Read PubSub messages",
                            PubsubIO.readStrings().fromSubscription(
                                    NestedValueProvider.of(
                                            options.getSubscriptionID(),
                                            input -> "projects/ozone-analytics-dev/subscriptions/" + input)
                            )
                    )

2.Created template

3.Running template with valid and existing subscriptionID

4.Seeing an error while creating dataflow job (I think that job didn't even start and this error is through some pre validation cause I can run this flow locally or normally with DataflowRunner but not via template) enter image description here

Solution

  • After analyse - this is rather Google Dataflow bug which only is related to template creation validation (because every other way of Dataflow run is behaving correctly)

    I've created issue for that https://issuetracker.google.com/u/1/issues/157584222

    And for now it is the answer for that problem