Search code examples
environment-variablesgcloudgoogle-cloud-composer

Setting Cloud Composer Environment Variables with Special Characters via gcloud


I'm trying to figure out how to create a Google Cloud Composer environment via gcloud and set environment variables with some special characters.

Example environmental variable: API_URL=https://stuff.api.thing.com

Example of gcloud cmd I'm using:

gcloud composer environments create new-env-1 --location=us-central1 --airflow_configs=[core-dags_are_paused_at_creation=True] --env-variables=[API_URL=https://stuff.api.thing.com,env_type=dev] --node_count=3 --python-version=3 --image-version=composer-1.10.0-airflow-1.10.6 --network=projects/backcountry-data-team/global/networks/default

When running the example above, I get the following error:

ERROR: (gcloud.composer.environments.create) argument --env-variables: Bad value [[API_URL]: Only upper and lowercase letters, digits, and underscores are allowed. Environment variable names may not start with a digit.

This error is expected based on the docs.

I am, however, able to set environmental variables like the example if I go through the GCP Console. So shouldn't there be a way to do this via gcloud?


Solution

  • For better visibility for the community, I'm making the comments from @DazWilkin to an answer.

    The gcloud command is expecting a comma separated "list" of key-value pairs; however, this differs to a list object on a programming language. In Python, for example, that list object would be something like this:

    [1,2,3]
    

    Instead the format expected for the gcloud tool is:

    ... the-flag=key1=value1,key2=value2
    

    Which can also be seen on the documentation (under gcloud tab) as follows:

    gcloud composer environments update ENVIRONMENT_NAME \
      --location LOCATION \
      --update-env-variables=KEY=VALUE,KEY=VALUE...
    

    And as @DazWilkin states, this means that you don't need to include the [ and ] around the key-value pairs.

    Leaving your command as next:

    gcloud composer environments create new-env-1 --location=us-central1 \
      --airflow_configs=core-dags_are_paused_at_creation=True \
      --env-variables=API_URL=https://stuff.api.thing.com,env_type=dev \
      --node_count=3 --python-version=3 --image-version=composer-1.10.0-airflow-1.10.6 \
      --network=projects/backcountry-data-team/global/networks/default