Search code examples
apache-kafkaconfluent-platform

Kafka Raft SASL_SSL client failing to initialize

I am struggling to get Kafka Raft running in my environment. I have tried multiple security protocols SASL_SSL, PLAINTEXT, SASL_PLAINTEXT -- The brokers cannot seem to get connected.

We do have a SELF SIGNED certificate which I've seen might be a concern for KRaft?

The other issue I've seen this error as well:

Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config.

This seems related to a Kerberos error but we are not running Kerberos in our environment.

Using the latest version of the open source confluent platform.

Current Docker Config

sudo docker run \
  -d \
  --name "kafka" \
  --restart always \
  --log-opt max-size=100k \
  --log-opt max-file=20 \
  -p 9092:9092 \
  -p 9093:9093 \
  -p 9094:9094 \
  -p 9095:9095 \
  -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_NODE_ID="1" \
  -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \
  -e KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER" \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@broker1:9095,2@broker2:9095,3@broker3:9095,4@broker4:9095,5@broker5:9095 \
  -e KAFKA_LISTENERS="CLIENTS://:9092,FAILOVER://:9093,INTERBROKER://:9094,CONTROLLER://:9095" \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="CLIENTS:SASL_SSL,FAILOVER:SASL_SSL,INTERBROKER:SASL_SSL,CONTROLLER:SASL_SSL" \
  -e KAFKA_INTER_BROKER_LISTENER_NAME="INTERBROKER" \
  -e KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL="SASL_SSL" \
  -e KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/secrets/kafka_server_jaas.conf" \
  -e KAFKA_ADVERTISED_LISTENERS="CLIENTS://:9092,FAILOVER://<ip>:9093,INTERBROKER://<ip>:9094" \
  -e KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN \
  -e KAFKA_SASL_ENABLED_MECHANISMS=PLAIN \
  -e KAFKA_AUTHORIZER_CLASS_NAME=com.company.kafka.security.authorization.CustomAuthorizer \
  -e KAFKA_HEAP_OPTS="-Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" \
  -e KAFKA_SSL_KEYSTORE_FILENAME=kafka.server.keystore.nonprod.jks \
  -e KAFKA_SSL_KEYSTORE_CREDENTIALS=credentials \
  -e KAFKA_SSL_KEY_CREDENTIALS=credentials \
  -e KAFKA_SSL_TRUSTSTORE_FILENAME=kafka.server.truststore.nonprod.jks \
  -e KAFKA_SSL_TRUSTSTORE_CREDENTIALS=credentials \
  -e JMX_PORT=9999 \
  -e KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" \
  -e KAFKA_SSL_CLIENT_AUTH=none \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=2 \
  -e KAFKA_MIN_INSYNC_REPLICAS=2 \
  -e KAFKA_DEFAULT_REPLICATION_FACTOR=3 \
  -e KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/etc/kafka/log4j/log4j.properties \
  -e LDAP_ENVIRONMENT="stage" \
  -e CONFLUENT_METRICS_ENABLE=0 \
  -e KAFKA_AUTO_CREATE_TOPICS_ENABLE="false" \
  -e KAFKA_CONTROLLED_SHUTDOWN_ENABLE="true" \
  -e KAFKA_REPLICA_LAG_TIME_MAX_MS="100000" \
  -e KAFKA_NUM_IO_THREADS="16" \
  -e KAFKA_NUM_NETWORK_THREADS="8" \
  -e KAFKA_REPLICA_FETCH_WAIT_MAX_MS="5000" \
  -e KAFKA_CONNECTION_FAILED_AUTHENTICATION_DELAY_MS=500000 \
  -v "/webApps/kafka/logs:/var/log/kafka" \
  -v "/webApps/kafka/data:/var/lib/kafka/data" \
  -v "/webApps/kafka/secrets:/etc/kafka/secrets" \
  -v "/webApps/kafka/log4j:/etc/kafka/log4j" \
  --network="host" \
  --user=0 \
  --ulimit nofile=1048576:1048576 \
  "kafka-broker:v7.4.3"

jaas_config

KafkaServer {
    com.company.kafka.security.authorization.CustomAuthorizer
    username="admin"
    user_admin="admin_pw"
    password="admin_pw";
};
KafkaClient {
    com.company.kafka.security.authorization.CustomAuthorizer
    username="admin"
    user_admin="admin_pw"
    password="admin_pw";
};

Logs:

[2024-01-11 18:03:22,795] WARN [RaftManager nodeId=1] Error connecting to node broker2:9095 (id: 2 rack: null) (org.apache.kafka.clients.NetworkClient)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348)
    at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
    at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:992)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1(InterBrokerSendThread.scala:103)
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1$adapted(InterBrokerSendThread.scala:99)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
    at kafka.common.InterBrokerSendThread.sendRequests(InterBrokerSendThread.scala:99)
    at kafka.common.InterBrokerSendThread.pollOnce(InterBrokerSendThread.scala:73)
    at kafka.common.InterBrokerSendThread.doWork(InterBrokerSendThread.scala:94)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
    at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:239)
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
    ... 13 more
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism SASL_SSL
Solution

  • The issue was that the environment variables for

    KAFKA_SSL_TRUSTSTORE_FILENAME KAFKA_SSL_KEYSTORE_FILENAME

    were changed to:

    KAFKA_SSL_TRUSTSTORE_LOCATION KAFKA_SSL_KEYSTORE_LOCATION

    The correct configuration is below:

    sudo docker run \
  -d \
  --name "kafka" \
  --restart always \
  --log-opt max-size=100k \
  --log-opt max-file=20 \
  -p 9092:9092 \
  -p 9093:9093 \
  -p 9094:9094 \
  -p 9095:9095 \
  -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_NODE_ID="1" \
  -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \
  -e KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER" \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@broker1:9095,2@broker2:9095,3@broker3:9095,4@broker4:9095,5@broker5:9095 \
  -e KAFKA_LISTENERS="CLIENTS://:9092,FAILOVER://:9093,INTERBROKER://:9094,CONTROLLER://:9095" \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="INTERBROKER:SASL_SSL,CONTROLLER:SASL_SSL" \
  -e KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN \
  -e KAFKA_SASL_ENABLED_MECHANISMS=PLAIN \
  -e KAFKA_SASL_ENABLED_MECHANISMS="PLAIN" \
  -e KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL="PLAIN" \
  -e KAFKA_LISTENER_NAME_CONTROLLER_SASL_ENABLED_MECHANISM="SASL_SSL" \
  -e KAFKA_ADVERTISED_LISTENERS="CLIENTS://:9092,FAILOVER://<ip>:9093,INTERBROKER://<ip>:9094" \
  -e KAFKA_AUTHORIZER_CLASS_NAME=com.company.kafka.security.authorization.CustomAuthorizer \
  -e KAFKA_HEAP_OPTS="-Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" \
  -e KAFKA_SSL_KEYSTORE_FILENAME=kafka.server.keystore.nonprod.jks \
  -e KAFKA_SSL_KEYSTORE_CREDENTIALS=credentials \
  -e KAFKA_SSL_KEY_CREDENTIALS=credentials \
  -e KAFKA_SSL_TRUSTSTORE_FILENAME=kafka.server.truststore.nonprod.jks \
  -e KAFKA_SSL_TRUSTSTORE_CREDENTIALS=credentials \
  -e JMX_PORT=9999 \
  -e KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" \
  -e KAFKA_SSL_CLIENT_AUTH=none \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=2 \
  -e KAFKA_MIN_INSYNC_REPLICAS=2 \
  -e KAFKA_DEFAULT_REPLICATION_FACTOR=3 \
  -e KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/etc/kafka/log4j/log4j.properties \
  -e LDAP_ENVIRONMENT="stage" \
  -e CONFLUENT_METRICS_ENABLE=0 \
  -e KAFKA_AUTO_CREATE_TOPICS_ENABLE="false" \
  -e KAFKA_CONTROLLED_SHUTDOWN_ENABLE="true" \
  -e KAFKA_REPLICA_LAG_TIME_MAX_MS="100000" \
  -e KAFKA_NUM_IO_THREADS="16" \
  -e KAFKA_NUM_NETWORK_THREADS="8" \
  -e KAFKA_REPLICA_FETCH_WAIT_MAX_MS="5000" \
  -e KAFKA_CONNECTION_FAILED_AUTHENTICATION_DELAY_MS=500000 \
  -v "/webApps/kafka/logs:/var/log/kafka" \
  -v "/webApps/kafka/data:/var/lib/kafka/data" \
  -v "/webApps/kafka/secrets:/etc/kafka/secrets" \
  -v "/webApps/kafka/log4j:/etc/kafka/log4j" \
  --network="host" \
  --user=0 \
  --ulimit nofile=1048576:1048576 \
  "kafka-broker:v7.4.3"