Search code examples
pythongoogle-bigqueryairflowgoogle-cloud-dataflow

Schedulling for Cloud Dataflow Job

So, I already finish to create a job in Dataflow. This job to process ETL from PostgreSQL to BigQuery. So, I don't know to create a schedulling using Airflow. Can share how to schedule job dataflow using Airflow?

Thank you

Solution

  • You can schedule dataflow batch jobs using Cloud Scheduler (fully managed cron job scheduler) / Cloud Composer (fully managed workflow orchestration service built on Apache Airflow).

    To schedule using Cloud Scheduler refer Schedule Dataflow batch jobs with Cloud Scheduler

    To schedule using Cloud Composer refer Launching Dataflow pipelines with Cloud Composer using DataflowTemplateOperator.

    For examples and more ways to run Dataflow jobs in Airflow using Java/Python SDKs refer Google Cloud Dataflow Operators