Search code examples
google-cloud-platformairflowgoogle-cloud-composer

GCP Airflow connection by using secret manager

I am trying to add Airflow connection for GCP cloud(SA key should be fetched from secret manager) but in my Airflow UI(version 2.1.4) I couldn't find option for adding by using secret manager. Is it because of version problem?

Image

If so can we add the airflow connection (by using secret manager) via command line (gcloud) or via programmatically to add it.

I tried via command line but it throws below error:

gcloud composer environments run project_id --location europe-west2 connections add --  edw_test    --conn-type=google_cloud_platform    --conn-extra '{"extra__google_cloud_platform__project": "proejct", "extra__google_cloud_platform__key_secret_name": "test_edw","extra__google_cloud_platform__scope": "https://www.googleapis.com/auth/cloud-platform"}'
kubeconfig entry generated
Unable to connect to the server i/o timeout
ERROR: (gcloud.composer.environments.run) kubectl returned non-zero status code.
Solution

  • I have upgraded both composer and airflow version which paved the way for creating the airflow connection by keeping the keys in secret manager