Is there withFormatFunction equivalent in Apache Beam Python SDK?

Question

I'm passing a PCollection of dictionary to WriteToBigQuery class. However, some fields of the dictionary aren't meant to be written to BigQuery tables. They're important to decide the table name for the element (in streaming mode). This is done by passing callable in the table parameter. Is this possible to do in Beam Python? This is possible in Java SDK through withFormatFunction of BigQueryIO. Cheers.

Answer 1

There's not currently an equivalent of withFormatFunction in Beam Python. If you have a fixed set of output tables, you can have separate WriteToBigQuery transforms for each one, and branch earlier in the pipeline. You could potentially also make a PCollection of objects of a type that acts like a dict (containing the payload), but also has fields on it that the table name callable can read