Creating IAM members in GCP throws error The size of the policy is too large

Question

When I try to create a new IAM user in GCP, I get the following error

Request "Create IAM Members roles/dns.admin serviceAccount:dns-mngt-sa-prod@pprojectId.iam.gserviceaccount.com 
for \"project \\\"projectId\\\"\"" returned error:
 Error applying IAM policy for project "projectId": Error setting IAM policy for project "projectId": googleapi: Error 400:
 The size of the policy is too large.
 Consider removing or merging some of the bindings or if using conditions remove 
any lengthy conditions., badRequest

Above shown is the output from terraform apply command. When I try to use gcloud cli, I get a similar output as well When I try to create it through GCP console, it just shows "backend error"

Answer 1

Most likely you hit one of the quotas of IAM. All the bindings you specified in Terraform are uploaded every time you perform an update or create. As the error says you are probably creating a lot of bindings that could be merged together. I suggest trying to find which quota limit you are reaching and why, here are some debugging commands:

Number of iam bindings:

gcloud projects get-iam-policy $PROJECT_ID --format=json | jq '.bindings | length'

Total size of the bindings:

gcloud projects get-iam-policy $PROJECT_ID --format=json | jq '.bindings' > iam.json
ls -l iam.json

Occurence of each member in all the bindings:

gcloud projects get-iam-policy $PROJECT_ID --format=json | jq ".bindings[].members" | jq -s flatten

Then, you could reduce the number of bindings by merging. For example, this binding:

{
  "bindings": [
    {
      "members": [
        "user:user-a@example.com"
      ],
      "role": "roles/owner"
    },
    {
      "members": [
        "user:user-b@example.com"
      ],
      "role": "roles/owner"
    }
  ],
  "etag": "BwUjMhCsNvY=",
  "version": 1
}

Could be merged to this:

{
  "bindings": [
    {
      "members": [
        "user:user-b@example.com",
        "user:user-a@example.com"
      ],
      "role": "roles/owner"
    }
  ],
  "etag": "BwUjMhCsNvY=",
  "version": 1
}