Why is @google-cloud/profiler throwing 'permission_denied' errors when running inside a GKE instance?

Question

I have been following the instructions on https://cloud.google.com/profiler/docs/profiling-nodejs#gke to use the gcloud profiler for my nodejs application:

I added RUN npm install @google-cloud/profiler to my dockerfile. And I added

require('@google-cloud/profiler').start({
    serviceContext: {
        service: 'your-service',
        version: '1.0.1',
    },
});

to my app.js file.

I then ran the container on a GKE cluster and got the following error:

@google-cloud/profiler Failed to create profile, waiting 8.6s to try again: Error: generic::permission_denied

By my understanding of the documentation I do not need explicit authentication or permissions to create profiles when running the code from within a gcloud hosted instance. The error itself isn't very helpful and I am a bit out of my depth here. I already tried if creating the cluster with --(autoprovisioning-)scopes "https://www.googleapis.com/auth/cloud-platform" might do the trick, but had no luck either.

Any ideas what might be the issue here?

Answer 1

You're correct that the documentation suggests you don't need credentials but I think that you do (and that the documentation is incomplete).

When an app (e.g. Profiler) uses Google's Application Default Credentials and runs on e.g. Compute Engine (App Engine, Cloud Run etc.) it is able to get credentials automatically from the environment (on GCP using Metadata service; locally using by the developer exporting GOOGLE_APPLICATION_CREDENTIALS to a key).

On GKE, I think this isn't true (unless Google's doing some magic somewhere) and that the app won't be able to access the credentials automatically unless you represent them in GKE. I think (!?) you'll need to create a Service Account and:

• either create a Kubernetes Secret from the Service Account's key
• or using GKE's Workload Identity

NOTE Workload Identity is a neat feature but it's only useful if you only ever plan to use GKE (and not some other Kubernetes implementation).

Before proceeding, let's see whether other folks reply to tell me that I'm wrong.

I wrote about using what was then called Stackdriver Profiler for GKE deployed apps. The article is dated but it should provide an outline of what you need to do.