I have two tables, both on Bigquery:
Every day, I want to run a job (with Cloud Composer) that read ONLINE table and insert new employees or/and update old ones. This upsert
operation is called MERGE
on Bigquery. I noticed that the Dataflow SQL doesn't allow to run job on Bigquery with the MERGE statement.
Is there an Airflow Operator (about Bigquery) that I can use instead of DataflowStartSqlJobOperator
?
If yes: is it right to i) delete rows from EMPLOYEE if ID is in ONLINE; ii) add ONLINE rows into EPLOYEE?
Solved. You can use the BigQueryInsertJobOperator
.
Here there's a guide on how to use it.