I've been creating a composer environment using the console interface, and then setting three variables using the cli (using examples from the QuickStart doc):
gcloud composer environments run etl --location europe-west1 variables -- --set gcp_project projectname
gcloud composer environments run etl --location europe-west1 variables -- --set gcs_bucket gs://bucketname
gcloud composer environments run etl --location europe-west1 variables -- --set gce_zone europe-west1-b
This doesn't set the bucket for some reason, but all three variables appear under Admin > Variables
I was hoping to combine all this into one step, and set the composer bucket too - so built the following CLI command:
gcloud composer environments create etl --location=europe-west1 --env-variables=gcp_project=projectname,gcs_bucket=gs://bucketname,gce_zone=europe-west1-b --service-account=serviceemail --zone=europe-west1-b
But this doesn't set the bucket and doesn't even create the variables.
How do I kick off a composer environment from the CLI with the bucket name and variables set?
Composer doesn't support user-provided GCS bucket, each newly created environment comes with its own GCS bucket.