GKE Autopilot Pod permission error when trying to publish message on GCP Pub/Sub

Question

I have a GKE Autopilot cluster with one service to publish a message to GCP Pub/Sub topic for testing.

I have created a KSA for the deployment, have used GCP Workload Identity Management to authorize the KSA to act as a GCP Service Account (GSA). Then I gave the GSA the Pub/Sub editor role.

The following commands are what I used:

kubectl create serviceaccount KSA_NAME

gcloud iam service-accounts add-iam-policy-binding GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[KSA_NAME]"

But even after doing all of this, I receive the following error when I try to publish a message to the topic.

ERROR:google.cloud.pubsub_v1.publisher._batch.thread:Failed to publish 1 messages.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 65, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.PERMISSION_DENIED
        details = "User not authorized to perform this action."
        debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.192.42:443 {grpc_message:"User not authorized to perform this action.", grpc_status:7, created_time:"2022-11-01T10:27:21.972013149+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 272, in _commit
    timeout=self._commit_timeout,
  File "/usr/local/lib/python3.7/site-packages/google/pubsub_v1/services/publisher/client.py", line 613, in publish
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 99, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 User not authorized to perform this action.

Here is the concerned code:

from flask import Flask, request, jsonify
app = Flask(__name__)

@app.route('/new_pub/<pages>')
def new_pubsub(pages: int):
    publish_message(pages)
    return json.dumps({'success': True}), 200, {
        'ContentType': 'application/json'}



from google.cloud import pubsub_v1
import json
def publish_message(data):
    d = {"message": data}
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path("MY_PROJECT", "TOPIC_NAME")
    publisher.publish(topic_path, json.dumps(d).encode('utf-8'))

Answer 1

Found the issue.

I missed to annotate the KSA with the email address of the IAM service account.

After these two commands,

kubectl create serviceaccount KSA_NAME

gcloud iam service-accounts add-iam-policy-binding GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "GSA_NAME:PROJECT_ID.svc.id.goog[KSA_NAME]"

I had to run this command:

kubectl annotate serviceaccount KSA_NAME \
    --namespace default \
    iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com