Deployment Manager Failed to Update Bucket Name

Question

I'm using the exact same template as the example template. I first deploy the template with bucket name X:

imports:
  - path: templates/gcs_bucket/gcs_bucket.py
    name: gcs_bucket.py

resources:
  - name: X
    type: gcs_bucket.py
    properties:
      name: X
      location: us-east1
      versioning:
        enabled: True
      labels:
        env: development

gcloud deployment-manager deployments create s3-sample --config gcs_bucket.yaml

which runs successfully.

Next, I tried to deploy using the same template but changing the bucket name to: Y

imports:
  - path: templates/gcs_bucket/gcs_bucket.py
    name: gcs_bucket.py

resources:
  - name: Y
    type: gcs_bucket.py
    properties:
      name: Y
      location: us-east1
      versioning:
        enabled: True
      labels:
        env: development

With the update command

gcloud deployment-manager deployments create s3-sample --config gcs_bucket.yaml

It failed with

Waiting for update [operation-1596317964460-5abd7bd3863a0-46f678bb-c6e8f7f7]...failed.                                                                                                                                                        
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1596317964460-5abd7bd3863a0-46f678bb-c6e8f7f7]: errors:
- code: RESOURCE_ERROR
  location: /deployments/s3-sample/resources/X
  message: '{"ResourceType":"gcp-types/storage-v1:buckets","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"XXX@cloudservices.gserviceaccount.com
    does not have storage.buckets.get access to the Google Cloud Storage bucket.","reason":"forbidden"}],"message":"XXX@cloudservices.gserviceaccount.com
    does not have storage.buckets.get access to the Google Cloud Storage bucket.","statusMessage":"Forbidden","requestPath":"https://storage.googleapis.com/storage/v1/b/X","httpMethod":"GET","suggestion":"Consider
    granting permissions to XXX@cloudservices.gserviceaccount.com"}}'

But the bucket Y was created successfully and the bucket X was not deleted? What did I do wrong? I tried the update command again after deleting the X bucket, and it worked.

Answer 1

In Cloud Storage, objects are immutable, which means that an uploaded or created object cannot change through its storage lifetime. This means that buckets can't be renamed.

Cloud Storage > Doc > Bucket naming guidelines > Bucket name requirements:

• A bucket name can only be assigned during creation. You cannot change the name of an existing bucket. Instead, you should create a new bucket with the desired name and move the contents from the old bucket to the new bucket. See Moving and Renaming Buckets for a step-by-step guide.

Cloud Storage > Doc > Moving and renaming buckets:

When you create a bucket, you permanently define its name. However, you can effectively move or rename your bucket:

• If there is no data in your old bucket, delete the bucket and create another bucket with a new name.

According to the error message, the second deployment does not have problems with the creation of a new bucket Y, but lacks permissions on the previously created bucket X.

By default the Google APIs Service Agent account used by Deployment Manager is granted the Project Editor role:

CloudShell:$ gcloud projects get-iam-policy my-project --flatten="bindings[].members" --format='table(bindings.role)' --filter="bindings.members:111111111111@cloudservices.gserviceaccount.com"
  ROLE
  roles/editor
CloudShell:$ gcloud iam roles describe roles/editor | grep storage.buckets
- storage.buckets.create
- storage.buckets.delete
- storage.buckets.list

As you can see, the Project Editor role includes the create and delete permissions on buckets, but not the get. That is why the first deployment goes Ok and creates a new bucket X. But the second deployment fails because it can't delete the previous bucket X, whereas it succeeds in creation of the new bucket Y. This also explains why the second deployment succeeds when you delete the bucket X manually.

To get this working, you should follow the documentation on the cloud-foundation-toolkit.
GitHub > GoogleCloudPlatform / cloud-foundation-toolkit > Google Cloud Storage Bucket:

Prerequisites

• Grant the storage.admin IAM role to the Deployment Manager service account

The Storage Admin role includes all storage.buckets.* roles, including get.
Cloud IAM > Doc > Understanding roles > Predefined roles > Cloud Storage roles: