Search code examples
dockergoogle-compute-enginegcloudgoogle-cloud-mlgoogle-container-os

Docker container for google cloudML on compute engine - authenticating for mounting bucket

I have been working with google's machine learning platform, cloudML.

Big picture: I'm trying to figure out the cleanest way to get their docker environment up and running on google compute instances, have access to the cloudML API and my storage bucket.

Starting locally, I have my service account configured

C:\Program Files (x86)\Google\Cloud SDK>gcloud config list
Your active configuration is: [service]

[compute]
region = us-central1
zone = us-central1-a
[core]
account = [email protected]
disable_usage_reporting = False
project = api-project-773889352370

I boot a compute instance with the google container image family

gcloud compute instances create gci --image-family gci-stable --image-project google-containers --scopes [email protected]="https://www.googleapis.com/auth/cloud-platform"

EDIT: Need to explicitly set scope for communicating with cloudML.

I can then ssh into that instance (for debugging)

gcloud compute ssh benweinstein2010@gci

On the compute instance, I can pull the cloudML docker from GCR and run it

docker pull gcr.io/cloud-datalab/datalab:local
docker run -it --rm  -p "127.0.0.1:8080:8080" \
  --entrypoint=/bin/bash \
  gcr.io/cloud-datalab/datalab:local

I can confirm I have access to my desired bucket. No credential problems there

root@cd6cc28a1c8a:/# gsutil ls gs://api-project-773889352370-ml
gs://api-project-773889352370-ml/Ben/
gs://api-project-773889352370-ml/Cameras/
gs://api-project-773889352370-ml/MeerkatReader/
gs://api-project-773889352370-ml/Prediction/
gs://api-project-773889352370-ml/TrainingData/
gs://api-project-773889352370-ml/cloudmldist/

But when I try to mount the bucket

root@139e775fcf6b:~# gcsfuse api-project-773889352370-ml /mnt/gcs-bucket
Using mount point: /mnt/gcs-bucket
Opening GCS connection...
Opening bucket...
Mounting file system...
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1

stderr:
fusermount: failed to open /dev/fuse: Operation not permitted

It must be that I am required to activate my service account from within the docker container? I have had similar (unsolved issues elsewhere)

gcloud auth activate-service-account

I could pass docker a credentials .json file, but i'm not sure where/if gcloud ssh passes those files to my instance?

I have access to cloud platform more broadly, for example I can post a request to the cloudML API.

gcloud beta ml predict --model ${MODEL_NAME} --json-instances images/request.json > images/${outfile}

which succeeds. So some credentials are being passed.I guess I could pass it to compute engine, and then from the compute engine to the docker instance? It feels like i'm not using the tools as intended. I thought gcloud would handle this once I authenticated locally.

Solution

  • This was a docker issue, not a gcloud permissions issue. Docker needs to be run as --privileged to allow fuse to mount.