Trigger a dataflow job deployed through Cloud Run on object creation in GCP Storage Bucket

Question

I have created a dataflow pipeline which read a file from GCS bucket and process it. It is working when I execute the job from my local.

I deployed the dataflow job in Cloud Run with trigger on storage.object.create.

But when I upload any file in GCS bucket, no trigger message shows in the log or dataflow job not executed.

Trigger config

Ingress:Allow traffic
Authentication:Allow authentication
Event source:Cloud Storage
Event type:google.cloud.audit.log.v1.written
Create time:2021-02-12 (16:05:25)
Receive events from:All regions (global)
Service URL path:/
Service account:[email protected]
Service name:storage.googleapis.com
Method name:storage.objects.create

What am I missing here? Please suggest.

Answer 1

The reason why your Cloud Run service isn't triggered is because there might be no audit logs written whenever an object is created/uploaded to your bucket. An Eventarc trigger is initiated whenever an event is written on Audit logs and by default, Cloud Storage is disabled:

The solution is to enable Audit Logs for Cloud Storage. It can be done two ways:

1. Enable it on the first time you create an Eventarc trigger.
2. Or go to IAM & Admin > Audit logs and make sure that all fields are checked for Cloud Storage:

---

As a reference, Audit logs can be seen on Home > Activity, here's an example: