GKE 1.4.5 (master and nodes) on gci cannot access cloud storage

Question

I am attempting to migrate our GKE cluster (running 1.4.5) from container-vm to gci. I am using the migration guide at https://cloud.google.com/container-engine/docs/node-image-migration

When I get my deployment on a gci node, I cannot access cloud storage, I use the google-cloud java 0.5.0 library (same issue with 0.4.0); the error I get is an insufficient rights, there are no problems on the container-vm node.

Here are my cluster permissions :

User info                  Enabled
Compute                    Read Write
Storage                    Read Write
Task queue                 Enabled
BigQuery                   Enabled
Cloud SQL                  Enabled
Cloud Datastore            Enabled
Stackdriver Logging API    Full
Stackdriver Monitoring API Full
Cloud Platform            Enabled
Bigtable Data             Read Write
Bigtable Admin            Full
Cloud Pub/Sub             Enabled
Service Control           Enabled
Service Management        Read Write
Stackdriver Trace         Disabled
Cloud Source Repositories Disabled

here is the error I get:

com.google.cloud.storage.StorageException: Insufficient Permission
at com.google.cloud.storage.spi.DefaultStorageRpc.translate(DefaultStorageRpc.java:202) ~[google-cloud-storage-0.5.0.jar:0.5.0]
at com.google.cloud.storage.spi.DefaultStorageRpc.create(DefaultStorageRpc.java:253) ~[google-cloud-storage-0.5.0.jar:0.5.0]
...
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
    {
       "code" : 403,
       "errors" : [ {
         "domain" : "global",
         "message" : "Insufficient Permission",
         "reason" : "insufficientPermissions"
       } ],
       "message" : "Insufficient Permission"
    }

Answer 1

I believe you have created the new node-pool through web UI(cloud console). If so, this is a known UI issue where you don't have an option to specify the scopes for the new node pool. Engineering team is already aware of this limitation and is working on a fix. With that said, if the scopes are not specified the default scopes are used for the new node pool. The workaround is either to upgrade the cluster using gcloud upgrade command with the gci image type or you can use gcloud to create the node pools and provide the necessary scopes for them as mentioned here.