Spring boot embedded kafka throws error BeanCreationException

Question

I have a test case with following configured kafka properties:

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = {Application.class})
@ActiveProfiles("test")
@EnableConfigurationProperties
@EmbeddedKafka(controlledShutdown = true, topics = {"topic1", "topic2", "topic3"})
class Name {}

Here is the build error i get when running the test case above:

Application run failed 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS
Caused by: java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS
    at kafka.server.Defaults$.<clinit>(KafkaConfig.scala:242)
    at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:961)
    at kafka.server.KafkaConfig.LogDirProp(KafkaConfig.scala)
    at org.springframework.kafka.test.EmbeddedKafkaBroker.afterPropertiesSet(EmbeddedKafkaBroker.java:322)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1853)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790)
    ... 72 common frames omitted

Here is the POM file with spring kafka & kafka-client dependency which is likely causing the error:

         <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>2.7.7</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka-clients</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>2.7.7</version>
            <scope>test</scope>
        </dependency>

Solution: Please refer to this link: https://docs.spring.io/spring-kafka/docs/current/reference/html/#update-deps

<dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>2.7.7</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
            <version>2.7.7</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.1</version>
            <classifier>test</classifier>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.13</artifactId>
            <version>2.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.13</artifactId>
            <version>2.8.1</version>
            <classifier>test</classifier>
            <scope>test</scope>
        </dependency>
    ```

Answer 1

That spring-Kafka version is not compatible with Apache Kafka client 3.0. You need still coming 2.8 and Spring Boot 2.6.

On the other hand you don’t need newer client even if you use newer broker . Although the story is about embedded testing, so I fully doubt you need to worry about any client, unless it is something transitive from Spring for Apache Kafka…