403 response with stackdrvier exporter on gcloud instance

Question

I spent my day on this and I juste can't see what I'm doing wrong and I can't find an answer to my issue in other posts, so I'm asking for help.

I have a compute instance with prometheus already install and a service account with role/viewer already configured for auto discovery and working.

I need to add to the metrics the stackdrivers monitoring metrics so I added the stackdriver exporter on the instance, still no problems the service is running and I can launch requests on it with prometheus.

The service is launch like this:

/usr/local/bin/stackdriver_exporter --google.project-id PROJECTID --monitoring.metrics-type-prefixes \
cloudsql.googleapis.com/,compute.googleapis.com/,vpn.googleapis.com/,loadbalancing.googleapis.com/,storage.googleapis.com/,redis.googleapis.com/

Nothing seems wrong there, I triple checked the project in case I'm turning blind and it seems good.

My problem is that I always end up with this error:

Nov 30 16:15:34 INSTANCENAME stackdriver_exporter[19149]: time="2018-11-30T16:15:34Z" leve
l=error msg="Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 403: Request had
 insufficient authentication scopes., forbidden" source="monitoring_collector.go:132"

I thought of a role issue, so I added monitoring viewer then monitoring admin and even role/owner but that didn't do anything.

I also tried setting the GOOGLE_APPLICATION_CREDENTIALS env variable to be certain of the account but still the same error.

I checked and Stackdriver API and Stackdriver monitoring API are enabled, I must be missing something but I just can't put my finger on it, any help would be appreciated.

Answer 1

Be aware that in gcloud the final permissions are a "mask" between the roles/permissions of the service account and the scopes of the instance.

When you set up an instance to run as a service account, the level of access the service account has is determined by the combination of access scopes granted to the instance and IAM roles granted to the service account.

So if you are running this with a service account with the role monitoring viewer in a instance without the scope https://www.googleapis.com/auth/monitoring.read you won't be able to read the monitor metrics.

1.- Check the scope of your instance with gcloud compute instances describe INSTANCE. You'll see something like this:

- email: 487724816353-compute@developer.gserviceaccount.com scopes: - https://www.googleapis.com/auth/monitoring.write - https://www.googleapis.com/auth/logging.write - https://www.googleapis.com/auth/devstorage.read_only Can you see the right scopes there?

2.- If not try to set the scopes that you need with: (You need to stop the instance for that)

gcloud compute instances set-service-account INSTANCE \ --service-account SERVICE_ACCOUNT@xxxxxxxx.iam.gserviceaccount.com \ --scopes cloud-platform

3.- The previous command allows full access to all the APIS in the instance as recommended in the gcloud best practices (https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#best_practices). And then you can control the final permissions just adding/removing roles to the service account.