Kafka Broker + Zookeeper + NiFi KafkaConsumer + Java Producer - SASL_PLAINTEXT and PLAIN Option

Question

For a project to simulate Produce and Consume a message through Kafka I am trying to setup a local environment with SpringBoot Java as a producer -> apache kafka 2.13 as Broker -> NiFi 1.22 (ConsumeKafka 2.0) as a Consumer.

I succeeded with PLAINTEXT option between Producer & Broker. But, NiFi ConsumeKafka library is having SASL Machanism is a mandatory field with atleast PLAIN as an option.

I have been going through apache kafka documentations and NiFi documentations and so far it is not successful.

Can someone help with what is the right configuration to be updated in Zookeeper (zookeeper.properties), Broker (server.properties) files and corresponding configurations for Java & NiFi!

Answer 1

I found the solution finally in a web page, but that is no more accessible. But, explaining the configurations whatever I used and tested in my environment.

Note: If you are going to run all applications in same environment, use separate bash sessions for each.

Configure Zookeeper

1.The first step is to assign KAFKA_HOME variable to your Kafka location. Take a fresh session and execute the following.

export KAFKA_HOME=<Your_Kafka_Dir>

2. Now we need to create a copy of the Zookeeper configuration, reason being that we want to clearly separate our SASL vs. SASL-less configuration files. To do this, run the following:

cp ${KAFKA_HOME}/config/zookeeper.properties {KAFKA_HOME}/config/zookeeper_sasl.properties

3. The next step is to update ${KAFKA_HOME}/config/zookeeper_sasl.propertiesso that it has the following contents appended to the very end:

#SASL

authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

requireClientAuthScheme=sasl

4. Prepare JAAS Config File

touch ${KAFKA_HOME}/config/zookeeper_jaas.config

5. Now we will add${KAFKA_HOME}/config/zookeeper_jaas.config with the following:

Server {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="zookeeper"

password="zookeeper-secret"

user_zookeeper="zookeeper-secret";

};

Now you are Good to Start Zookeeper

export KAFKA_HOME=$PATH && export KAFKA_OPTS="-Djava.security.auth.login.config=${KAFKA_HOME}/config/zookeeper_jaas.config" && sh ${KAFKA_HOME}/bin/zookeeper-server-start.sh config/zookeeper_sasl.properties

Configure Broker

1. Set KAFKA_HOME variable
2. Just like with ZooKeeper, we will create a copy of the Kafka Broker configuration file and update it for enabling SASL instead of modifying the original file.

cp ${KAFKA_HOME}/config/server.properties ${KAFKA_HOME}/config/server_sasl.properties 3. The next step is to update ${KAFKA_HOME}/config/server_sasl.properties so that it has the following contents appended to the very end:

#SASL

sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN security.inter.broker.protocol=SASL_PLAINTEXT listeners=SASL_PLAINTEXT://”your wsl ip”:9092 advertised.listeners=SASL_PLAINTEXT://”your wsl ip”:9092

4. Create and update JAAS Config

touch ${KAFKA_HOME}/config/server_jaas.config

KafkaServer {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="admin"

password="admin-secret"

user_admin="admin-secret"

user_consumer="consumer-secret"

user_producer="producer-secret";

};

Client {

org.apache.kafka.common.security.plain.PlainLoginModule required username="zookeeper"

password="zookeeper-secret";

};

Now good to start Kafka Broker.

export KAFKA_HOME= && export KAFKA_OPTS="-Djava.security.auth.login.config=${KAFKA_HOME}/config/zookeeper_jaas.config" && ${KAFKA_HOME}/bin/kafka-server-start.sh config/server_sasl.properties

Producer or Consumer Configuration