Search code examples
c#.netapache-kafkaaws-msksasl-scram

Connect from local machine to aws MSK

I followed "public access" to set up the configuration. I have two goals, Firstly, I want to create topic from local terminal by using this command line "/bin/kafka-topics.sh --create --bootstrap-server ZookeeperConnectString --replication-factor 3 --partitions 1 --topic ExampleTopicName", but it always return "the broker is not available". Secondly, I want to connect MKS from local .Net Application. However, it seams cannot connect to the MKS successfully.

This is my some configuration that attach on my MKS

  1. Create public subnet 172.31.0.0/20 and 172.31.16.0/20 and attach an Internet Gateway
  2. Close unauthenticated access control off and turn on SASL/SCRAM access-control methods. Besides, I attached an secret for this authentication and add allow.everyone.if.no.acl.found to false to cluster's configuration.
  3. Turn on public access
  4. Cluster configuration Cluster configuration
  5. Producer Configuration Producer Configuration
  6. Security Group Security Group

Does anyone can give me some advice or hints? I do some research that not sure I have to add listeners in my cluster configuration? Thanks for your time and consideration.

Solution

  • I was struggling with MSK, too. I finally got it working and maybe give some hints here:

    • according to the docs at AWS, only SCRAM-SHA-512 is supported, not -256
    • in the SG, I did add a rule for inbound traffic to accept from anywhere (0.0.0.0)

    Hope that helps.