Google Cloud DataStream to Bigquery template not able to sync data to big query
I am trying to design CDC pipeline to stream data from cloud SQL to BigQuery using DataStreams and Dataflow on GCP, the datastream part is working fine and I can see data being transferred to CloudStorage successfully in avro format.
When it comes to DataFlow, I am using DataFlow Template DataStream to BigQuery with the configuration in the screenshot
I can see the DataFlow job started and running with no errors in the log, yet I can't see any data transfer happening from Cloud Storage to BigQuery.
It looks to me there is something missing, which is the link between Cloud storage and Pub/Sub, I think it there should be a link to stream the data from GCS to Pub/Sub, and eventually DataFlow stream from Pub/Sub to BQ, no?
What I am missing here?
It was something missing from my side which is setting up the link between GCS and Pub/Sub using the blow command
gsutil notification create -f "none" -p "db/" -t "datastream" "gs://my-buk"
- Firebase Firestore: Accessing Never-Retrieved Data Offline
- Google recaptcha enterprise: Your default credentials were not found
- Firebase Admin SDK cant upload to storage bucket folder
- Accessing Firestore via Cloud Function
- Problem with executing asynchronous Cloud Function
- do you need to create a separate collection/document for reading an aggregated document with firebase cloud function
- httpsCallable is not a function when calling a Firebase function
- Google Cloud Eventarc: Receive events via Pub/Sub directly
- Android Studio's project gradle file changed?
- How to listen to button triggers using Firebase
- gcloud auth login throwing error: gcloud crashed (ConnectionError): HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded
- Google Cloud Build Error: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
- Terraform - GCP - Import project IAM member
- Choosing fields to return with Places API (new) client library
- Deploy to Update version of Gmail App Script Extension
- Getting random "User not found in file /etc/passwd" error on Cloud Run Job
- Change time format in GCP Logs Explorer
- Deploying to Cloud Run with a custom service account failed with iam.serviceaccounts.actAs error
- Deploying a pelican site to Google Cloud Run with Dockerfile not working
- Is there a good explanation of resource labels versus metric labels?
- Why do I set the resource type when writing the metric but not when creating a descriptor?
- google-github-actions/get-gke-credentials failed with: required "container.clusters.get" permission(s)
- GA4 Data Not Retained for More Than 60 Days Despite BigQuery Table Expiration Set to 'Never
- How to list all enabled API services for Google Cloud Platform project
- Google Cloud calendar API connection error
- Downloading a folder from cloud storage in GitHub action and move it inside a docker container is not working
- How to fix CloudRun error 'The request was aborted because there was no available instance'
- Google Cloud Build with Dockerfile and copy files
- Firebase CLI console: Cannot find module @google-cloud/tasks
- Does Cloud SQL have a query console or is CLI the only option?