Search code examples
google-cloud-platformterraformgoogle-cloud-storagegoogle-cloud-pubsubterraform-provider-gcp

How to properly assign publisher and subscriber roles to a dead-letter topic using terraform


I have a Cloud Storage bucket defined in terraform. The bucket has notifications enabled. The notifications are published to a Pub/Sub topic which is then consumed by a Pub/Sub subscription. I have added dead-lettering to the subscription so I can receive any unacknowledged messages. This is all working well apart from the dead-lettering. Looking at my GCP console, I have the following errors regarding publish/subscribe permissions

enter image description here

I can't figure out where the problem is. Below is my current setup and what I have tried so far.

data "google_storage_project_service_account" "gcs_account" {}

data "google_project" "project" {}

// The topic to which Cloud Storage events will be published to.
resource "google_pubsub_topic" "requests" {
  name = "example_name"
}

// After a specified number of failed messages, send to this topic
resource "google_pubsub_topic" "dead_letter" {
  name = "example_name"
}

// The bucket
resource "google_storage_bucket" "requests" {
  force_destroy               = true
  location                    = "EUROPE-WEST1"
  name                        = "example_name"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  lifecycle_rule {
    condition {
      age = 1
    }
    action {
      type = "Delete"
    }
  }
}

// Create a Google Cloud Storage event notification
resource "google_storage_notification" "request" {
  bucket         = google_storage_bucket.requests.name
  payload_format = "JSON_API_V1"
  topic          = google_pubsub_topic.requests.id
  event_types    = ["OBJECT_FINALIZE", "OBJECT_METADATA_UPDATE"]
  depends_on     = [
    google_pubsub_topic_iam_binding.request_binding,
    google_pubsub_topic_iam_binding.pubsub_sa_publish_deadletter_topic,
    google_pubsub_subscription_iam_binding.pubsub_sa_pull_request_sub
  ]
}

// The Google Cloud storage service account need to be able to publish an event to the requests topic.
resource "google_pubsub_topic_iam_binding" "request_binding" {
  topic   = google_pubsub_topic.requests.id
  role    = "roles/pubsub.publisher"
  members = [
    "serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"
  ]
}

// The Google Cloud storage service account need to be able to publish an event to the dead letter topic.
resource "google_pubsub_topic_iam_binding" "pubsub_sa_publish_deadletter_topic" {
  topic   = google_pubsub_topic.dead_letter.name
  role    = "roles/pubsub.publisher"
  members = [
    "serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}",
    "serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
  ]
}

resource "google_pubsub_subscription" "request" {
  name                       = "example_name"
  topic                      = google_pubsub_topic.requests.name
  message_retention_duration = "820s"
  ack_deadline_seconds       = 300
  retry_policy {
    maximum_backoff = "77s"
    minimum_backoff = "7s"
  }
  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter.id
    max_delivery_attempts = 7
  }
  expiration_policy {
    ttl = ""
  }
  push_config {
    oidc_token {
      audience              = "example_audience"
      service_account_email = "example@${var.SOME_PROJECT}.iam.gserviceaccount.com"
    }
    push_endpoint = "example_url"
  }
}

// The Google Cloud storage service account need to be able to pull events from the request topic.
resource "google_pubsub_subscription_iam_binding" "pubsub_sa_pull_request_sub" {
  subscription = google_pubsub_subscription.request.name
  role         = "roles/pubsub.subscriber"
  members      = [
    "serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}",
    "serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
  ]
}


Solution

  • For anyone with a similar issue. After reading https://stackoverflow.com/a/70253521/20262182, I switched from using google_pubsub_topic_iam_binding to google_pubsub_topic_iam_member and google_pubsub_subscription_iam_binding to google_pubsub_subscription_iam_member and that worked. Not sure if the iam_binding(Authoritative) was messing with some related permissions.