Forward messages from Amazon MSK to GCP pub/sub

Question

Is there any connector to connect AWS MSK with GCP pub/sub service? I found this article, but seems this one for on side Kafka. Any suggestions?

Answer 1

Amazon MSK stands for Amazon Managed Streaming for Apache Kafka.

In other words, MSK is Kafka, when you interact with it using Apache Kafka APIs - consumer, producer or admin.

The provided link leads to a sink and source connectors that can connect MSK (or any other Kafka) cluster and GCP Pub-Sub. Either sink or source connector should be deployed on Kafka Connect cluster, which needs to be in close proximity with hosting Kafka cluster. This can be any kind of self-managed Kafka Connect on EC2, or any container solution, as long as you know how to deal with. Alternatively, these connectors can run on MSK Connect, which is a managed Kafka Connect.

What you will need:

1. Ensure connectivity between VPC/subnets where Kafka Connect with connectors runs, to the GCP Endpoint
2. Start Kafka Connect cluster which is using MSK as a hosting Kafka cluster. Copy binaries of those connectors to a plugin dir defined in worker configuration. Under MSK Connect - create a custom plugin
3. Create connectors by using rest endpoint of self-managed kafka connect or by using MSK Connect API
4. Pray: Observe, monitor and troubleshoot until it's successfully pulling/pushing some data