Using Confluent Schema Registry with MSK

Question

Is it possible to integrate Confluent Schema Registry with AWS MSK? If you have done this before, can you please provide some pointers / blogs you followed to achieve it?

Answer 1

It is possible. My setup uses ec2 and docker.

1. Download the IAM auth jar if you are using IAM based auth

mkdir -p /usr/share/java/aws
wget -P /usr/share/java/aws https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
chmod -R 444 /usr/share/java/aws

2. Use confluent official docker image for schema registry

...

  schema-registry:
    image: confluentinc/cp-schema-registry:5.4.6-1-ubi8
    hostname: schema-registry
    container_name: schema-registry
    ports:
      - "8081:8081"
    volumes:
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/cp-base-new/aws-msk-iam-auth-1.1.1-all.jar
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/rest-utils/aws-msk-iam-auth-1.1.1-all.jar
    environment: # https://docs.confluent.io/platform/current/schema-registry/installation/config.html#schemaregistry-config
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_HOST_NAME: "${HOSTNAME}" # 
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "${BOOTSTRAP_BROKERS_SASL_IAM}"
      SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: "SASL_SSL"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM: "AWS_MSK_IAM"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG: "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

...

• HOSTNAME is your ec2 machine DNS name or IP, example ip-10-0-0-84.ec2.internal
• BOOTSTRAP_BROKERS_SASL_IAM is comma separated host1:port,host2:port urls. For port information see this

If you are using PLAINTEXT or SSL auth, last 4 environment variable changes. And you don't have to download iam auth jar

3. Configure source or sink connector with these property

...
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schemas.enable=true
value.converter.schema.registry.url=http://ip-10-0-0-84.ec2.internal:8081
value.converter.enhanced.avro.schema.support=true

Thats it.
Do open 8081 port in your security group of EC2 instance for MSK cluster

Resource:

• https://github.com/maikelpenz/kafka-aws-deployment
• https://maikelpenz.medium.com/building-a-kafka-playground-on-aws-part-3-streaming-database-events-and-querying-with-ksql-5f23978a0080

---

Alternate option I have tried is AWS Glue Schema registry But we had to use KSQL, and KSQL does't have 3rd party schema registry integration or custom SerDe Github issue