I currently have a scenario wherein I created deployment script (python) using the Deployment Manager, it runs perfectly well in a GCP sandbox (LA Playground) I tend to use to try things out. But when I tried running it on an actual GCP project with billing accounts etc., I get a permission error even though I'm using basically the same set of roles in the service account I created for it. I also am the project owner of that troubling GCP account.
The particular set of permission I'm having problems with is in creating project sinks. I always get the following error:
Error in Operation [operation-xxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxxx-9d487530]: errors:
- code: RESOURCE_ERROR
location: /deployments/structured-pipeline/resources/dataprep-bq-listener-sink
message: '{"ResourceType":"gcp-types/logging-v2:projects.sinks","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"The
caller does not have permission","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://logging.googleapis.com/v2/projects/redacted-gcp-project-name-dev/sinks","httpMethod":"POST"}}'
Here's the list of roles I have attached on the service account that is used to run the deployment:
As you may see, I already made them admin level just to make sure I'm not missing a required permission.
As per the official documentation, the specific permission needed for creating projects sinks is logging.sinks.create which is included in both Logging Admin & Logs Configuration Writer roles.
UPDATE#1:
Here's the expanded DM config of the troubled resources
- name: dataprep-bq-listener-sink
properties:
destination: pubsub.googleapis.com/projects/redacted-gcp-project-name-dev/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
filter: |-
resource.type="bigquery_dataset"
resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh"
protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"
outputVersionFormat: V2
sink: dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh
type: gcp-types/logging-v2:projects.sinks
- accessControl:
gcpIamPolicy:
bindings:
- members:
- $(ref.dataprep-bq-listener-sink.writerIdentity)
role: roles/pubsub.publisher
name: dataprep-bq-listener
properties:
labels:
client_namespace: redacted-client-name
topic: dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
type: pubsub.v1.topic
metadata:
dependsOn:
- dataprep-bq-listener-sink
UPDATE#2
I set my gcloud to use a service account for credentials (https://cloud.google.com/sdk/docs/authorizing#authorizing_with_a_service_account) which has the roles I stated above.
Running gcloud auth list clearly points to the service account I created/used.
ACTIVE ACCOUNT
my_redacted_email@company.com
cloud_user_p_2725ef46@linuxacademygclabs.com
cloud_user_p_2e3db20d@linuxacademygclabs.com
cloud_user_p_41b5121a@linuxacademygclabs.com
cloud_user_p_b57aaef8@linuxacademygclabs.com
cloud_user_p_bdb72060@linuxacademygclabs.com
cloud_user_p_c2f5d19a@linuxacademygclabs.com
cloud_user_p_c3c54122@linuxacademygclabs.com
cloud_user_p_c88350f1@linuxacademygclabs.com
cloud_user_p_d7702b8b@linuxacademygclabs.com
* service_account_used_in_dm@project_id.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
UPDATE#3
Running gcloud projects get-iam-policy <PROJECT_ID> results to the following list of IAM policies (I've replaced my email with "my_redacted_email@company.com" and the service account email I used for the DM with "service_account_used_in_dm@project_id.iam.gserviceaccount.com"; all the default default service accounts I retained)
bindings:
- members:
- user: my_redacted_email@company.com
role: roles/billing.projectManager
- members:
- serviceAccount:1008104628570@cloudbuild.gserviceaccount.com
role: roles/cloudbuild.builds.builder
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudbuild.iam.gserviceaccount.com
role: roles/cloudbuild.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcf-admin-robot.iam.gserviceaccount.com
role: roles/cloudfunctions.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudscheduler.iam.gserviceaccount.com
role: roles/cloudscheduler.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/compute.admin
- members:
- serviceAccount:service-1008104628570@compute-system.iam.gserviceaccount.com
role: roles/compute.serviceAgent
- members:
- serviceAccount:service-1008104628570@containerregistry.iam.gserviceaccount.com
role: roles/containerregistry.ServiceAgent
- members:
- serviceAccount:service-1008104628570@dataflow-service-producer-prod.iam.gserviceaccount.com
role: roles/dataflow.serviceAgent
- members:
- serviceAccount:service-1008104628570@trifacta-gcloud-prod.iam.gserviceaccount.com
role: roles/dataprep.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/deploymentmanager.editor
- members:
- serviceAccount:1008104628570-compute@developer.gserviceaccount.com
- serviceAccount:1008104628570@cloudservices.gserviceaccount.com
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
- user:my_redacted_email@company.com
role: roles/editor
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/iam.securityAdmin
- members:
- user:my_redacted_email@company.com
role: roles/iam.serviceAccountUser
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.admin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.configWriter
- members:
- user:my_redacted_email@company.com
role: roles/owner
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/pubsub.admin
- members:
- user:my_redacted_email@company.com
role: roles/resourcemanager.projectIamAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/serviceusage.serviceUsageAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/storage.admin
etag: BwW0MM3sGXk=
version: 1
As you may noticed the service account I'm using has bindings to the roles that I mentioned above at the project level.
UPDATE#4
Creating the sink through gcloud using the same service account results in a successful creation of the sink.
gcloud logging sinks create dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh pubsub.googleapis.com/projects/<project_id>/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh --log-filter='resource.type="bigquery_dataset" AND resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh" AND protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"' --project=<project_id>
UPDATE#5
@Kolban did note of the requirement of Deployment Manager for the [PROJECT_NUMBER]@cloudservices.gserviceaccount.com service account to the Editor role as per stated in their official documentation, now looking at UPDATE#4, it clearly shows that the said service account has the Editor role.
He also noted for possible mixing up of multiple cloud accounts (I might be running the command using the service account but under a different cloud account), thus, I executed the following commands: gcloud config get-value account & gcloud config get-value project which resulted to what I'm expecting which is the correct GCP project and Service Account pair.
The docs state:-
To create other Google Cloud resources, Deployment Manager uses the credentials of the Google APIs service account to authenticate to other APIs. The Google APIs service account is designed specifically to run internal Google processes on your behalf. The service account is identifiable using the email:
[PROJECT_NUMBER]@cloudservices.gserviceaccount.com
In your case, your 1008104628570@cloudservices.gserviceaccount.com Service Account is bound to the roles\editor role at project level. However, the permission you require (logging.sinks.create) isn't contained within that legacy Editor role.
Can you try additionally granting the 1008104628570@cloudservices.gserviceaccount.com Service Account the Logging Admin role (roles/logging.admin) and see if that helps?