Google Cloud Storage To Google Cloud SQL (Postgres) Operator in Airflow (or Composer)

Question

I am trying to load data from a CSV file in GCS, but there is no predefined operator that does this in Airflow.

I built a simple operator using a PSQL hook and a GCS file reader, but I'm wondering if there is a better solution for this, as right now the way the custom operator workes is running on a loop, row by row, a series of "INSERT INTO" statements with the open GCS file.

Answer 1

Yes there is no operator to insert data from GCS into CLoud SQL, but you can use the CloudSqlHook, to import the GCS file.

Here you find an example for body, which is a dict contains your file rows, if your file is too big, you can import it in batchs (1k-10K rows) which is much better than a loop with INSERT INTO.