Search code examples
pythonmysqlgoogle-bigqueryetlgoogle-cloud-dataflow

Load data from MySQL to BigQuery using Dataflow


I want to load data from MySQL to BigQuery using Cloud Dataflow. Anyone can share article or work experience about load data from MySQL to BigQuery using Cloud Dataflow with Python language?

Thank you


Solution

  • You can use apache_beam.io.jdbc to read from your MySQL database, and the BigQuery I/O to write on BigQuery.

    Beam knowledge is expected, so I recommend looking at Apache Beam Programming Guide first.

    If you are looking for something pre-built, we have the JDBC to BigQuery Google-provided template, which is open-source (here), but it is written in Java.