Search code examples
pythongoogle-cloud-dataflowapache-beam

Apache Beam DynamicDestinations Python equivalent


The Apache Beam Java SDK has a DynamicDestinations class that allows writing to different big query tables based on the input element. I don't see anything in the Python SDK that looks equivalent. Is there some class that allows writing to dynamically chosen destination tables in the Python SDK?


Solution

  • The Apache Beam Python SDK is still not as advanced as the Java SDK in the sense of variety of features, so it is true that you may see some missing features that are still only available in the Java SDK.

    As far as I know, and based on what I can find the Python SDK Beam's BigQuery IO documentation, there is currently no class available to specify dynamic BigQuery destinations, as the one that the Java SDK has available (Dynamic Destinations in Java SDK).

    I would suggest that you file a new issue as a feature request in the Apache Beam Jira issue tracker explaining why this feature would be a good addition to the Python SDK, and hopefully the developers will consider this option too.