How to change the Google Cloud Composer environments's service account after the environment has been created?
We have a cloud composer instance set up with the default compute engine service account as the environment's service account (set up by default). We have created a new service account to use instead in the cloud composer instance but we can't find an option to change the environment's service account in the Environment Configuration page. Other fields have "Edit" options at the end of their rows but we can't find the option to edit the service account field.
I found an old Google Group conversation from 2018 that said it's technically possible by creating a new node pool in GKE but this isn't recommended due to the likelihood that it will break the environment.
How do we swap out the old, default compute engine service account to our new service account?
Cloud Composer 2 uses two types of IAM service accounts:
- Cloud Composer Service Agent account
- Environment's service account
Normally the Cloud Composer default SA can't be changed in the configuration and it will be used to create the Composer instances and machines.
But you can set the environment's service account. This account will be used at runtime with Airflow scheduler and workers, to allows access to GCP resources from DAGs.
Unfortunately if you created a Composer instance with an environment's service account, you can't change it after.
If you need to change this SA, you have to destroy and recreate the Composer Cluster with this new Service Account.
If you have a tool like Terraform, it's easier to automate this process and recreate the Cluster with your configuration.
In the case of Composer cluster recreation, you can copy all your DAG folders from the previous bucket to the new with gsutil.