Search code examples
amazon-web-servicesgoogle-cloud-platformgoogle-bigqueryamazon-cloudwatchstackdriver

Is there an AWS equivalent to `Stackdriver Sinks to BigQuery`

I use a lot StackDriver sinks to BigQuery. It helps on keeping the Data and have them in a convenient queryable form. I am searching for the equivalent on AWS using CloudWatch but it seems only S3 is integrated. Any workarounds or should I code it?

Solution

  • This can be done with a workaround. It needs only one code component reading from S3 and posting to Stackdriver, then GCP sinks can pickup.

    Create a CloudWatch Sink to S3 bucket.

    The S3 bucket will be integrated to send SNS messages on object Creation.

    A process should listen to those SNS notifications and copy the contents of the bucket into StackDriver.

    Since the guarantees of SNS and Cloudwatch to S3 are there, a Lambda function on the SNS notification messages can make the solution more seamless.