Search code examples
google-cloud-platformgoogle-cloud-dataproc

Using non-default service account in Google Cloud dataproc

I'd like to create a dataproc cluster that runs under a non-default service account. The following works for a compute instance:

gcloud compute instances create instance-1 --machine-type "n1-standard-1" --zone "europe-west1-b" --scopes [email protected]="https://www.googleapis.com/auth/cloud-platform"

But the same --scopes argument fails when creating a dataproc instance:

gcloud dataproc clusters create --zone "europe-west1-b" --scopes [email protected]="https://www.googleapis.com/auth/cloud-platform" testdataproc12345

ERROR: (gcloud.dataproc.clusters.create) Invalid service account scope: '[email protected]=https://www.googleapis.com/auth/cloud-platform'

Is it possible to run dataproc under a non-default service account?

Solution

  • Unfortunately, at the moment there's no way to specify your custom service accounts using the normal "scopes and metadata"-mediated auth setup. This is a known feature request, however, so it should become available in a future Dataproc update.

    In the meantime, even though you can't disable the existence of the "storage read/write" scope with the default GCE service account when using Dataproc, you can make the Hadoop side use a particular service account via keyfiles by using the "Create Key" option under the IAM & Admin > Service accounts page to obtain a JSON keyfile for your service account, and then do two things:

    1. Add the following property at cluster creation time:

      --properties core:fs.gs.auth.service.account.json.keyfile=/etc/hadoop/conf/my-service-account.json

    2. Use an init action which copies your JSON keyfile to your nodes; note that this still means your JSON keyfile must be accessible to the GCE default service account as a reader, and anyone who has access to the GCS location of your JSON keyfile also has the ability to now act on behalf of that service account, so you still need to keep your project secure as necessary.

      #!/bin/bash
# Save this somewhere as gs://somepath/my-keyfile-setup.sh

gsutil cp gs://path/to/your/json/file/in/gcs/my=service-account.json \
    /etc/hadoop/conf/my-service-account.json

      And then apply that init action:

      gcloud dataproc clusters create --initialization-actions gs://somepath/my-keyfile-setup.sh ...