Search code examples
google-bigquerystreamingairflow

Airflow bigquery_to_bigquery operator, is it using streaming to insert data to destination table?


https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/bigquery_to_bigquery.html

Copies data from one BigQuery table to another.

When data copied from source table to destination table, is it using bigquery streaming to insert data to destination table?

In other words, if the streaming is used here. and if I just run this bigquery_to_bigquery write_truncate task, and when the streaming buffering is still not empty yet and I re-run this task again too soon, would there be problems?

Like when we do the write-truncate in streaming, https://cloud.google.com/bigquery/docs/error-messages#streaming


Solution

  • In looking at the documentation for airflow it looks like that calls the jobs api with a configuration of copy, which is different than the streaming api.