Search code examples
amazon-web-servicesaws-mskaws-msk-connect

Using multiple topics in AWS MSK Connect

I created a AWS MSK Connector with below configuration. The export to S3 bucket was successful.

Now how do I use wild card configuration for a stack with like 1000 plus topics in MSK cluster. As you can see below I cannot include names of each topic separated by comma. Wanted to know the syntax for wildcard name in the configuration code. I tried * already and did not work.

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=us-east-1
format.class=io.confluent.connect.s3.format.json.JsonFormat
flush.size=100
schema.compatibility=NONE
tasks.max=2
topics=MSKTutorialTopic
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
storage.class=io.confluent.connect.s3.storage.S3Storage
s3.bucket.name=name-of-bucket
Solution

  • After reading more in the documentation [1] it can be achieved by using regex as below.

        connector.class=io.confluent.connect.s3.S3SinkConnector
    s3.region=us-east-1
    format.class=io.confluent.connect.s3.format.json.JsonFormat
    flush.size=100
    schema.compatibility=NONE
    tasks.max=2
    topics.regex=MSK.*
    partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
    storage.class=io.confluent.connect.s3.storage.S3Storage
    s3.bucket.name=name-of-bucket

    [1] https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html