Sink all projects under an organisation into a bigtable

Question

I am building a similar approach to bring all the logs into BigQuery dataset. As per this https://dev.to/alvardev/gcp-cloud-logging-the-basis-45eh I am able to bring the project log details into the corresponding BigQuery. But how do we bring all the projects logs into a single Bigtable i.e for example lets say An organisation has 50 projects and with minimal configuration I want all the logs from 50 projects to come under a single Bigtable as consolidated project logs.

Is this possible in GCP, can someone please help me on this

Answer 1

Currently, the only supported destinations based on the documentation are Cloud Storage, Pub/Sub, BigQuery and another Cloud Logging Bucket . However I found this another approach where the architecture is: Cloud Logging -> Pub/Sub -> DataFlow -> BigTable.

Basically, you will create a sink and choose Pub/Sub topic as the destination then from there, you can stream your data from Pub/Sub to Bigtable using Dataflow. You can check this SO answer that you can use as reference on how to stream data from Pub/Sub to BT.