Search code examples
google-bigquerygoogle-cloud-data-fusion

Perform custom SQL query with Google Cloud Data Fusion

I have data pipelines that consist of multiple SQL queries being run against BigQuery tables, I would like to build these in Google Cloud Fusion, but I don't see an option to transform/select with custom SQL.

is this available, or am I misinterpreting the use cases for this tool?

Solution

  • There is now a native BigQuery Execute action that allows SQL queries to run as part of a Data Fusion Pipeline.

    enter image description here

    This job is an action, see below from the official documentation:

    Action plugins define custom actions that are scheduled to take place during a workflow but don't directly manipulate data in the workflow. For example, using the Database custom action, you can run an arbitrary database command at the end of your pipeline. Alternatively, you can trigger an action to move files within Cloud Storage.