Search code examples
spring-bootmavenspring-kafka

Maven dependency tree has older version than what is the transitive dependency pom of dependent library

I have a maven project which uses spring-kafka. If I run mvn dependency:tree I get:

....
[INFO] +- org.springframework.kafka:spring-kafka:jar:3.3.0:compile
[INFO] |  +- org.springframework:spring-messaging:jar:6.1.14:compile
[INFO] |  +- org.springframework:spring-tx:jar:6.1.14:compile
[INFO] |  +- org.springframework.retry:spring-retry:jar:2.0.10:compile
[INFO] |  \- org.apache.kafka:kafka-clients:jar:3.7.1:compile
[INFO] |     +- com.github.luben:zstd-jni:jar:1.5.6-3:runtime
[INFO] |     +- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] |     \- org.xerial.snappy:snappy-java:jar:1.1.10.5:runtime
....

This shows kafka-clients version 3.7.1.

If I look at the pom.xml of spring-kafka:3.3.0 it shows version 3.8.1 as a dependency:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.8.1</version>
  <scope>compile</scope>
</dependency>

So why is version 3.7.1 pulled in? I can provide the whole dependency:tree output.

There are only 2 dependencies which have kafka-clients as a transient: spring-kafka and spring-kafka-test. Both pull in 3.7.1 but yet have 3.8.1 as dependency versions in their pom.xml.

I've run mvn clean install -U, dependency:resolve -U and tried many other updates. I've tried explicitly setting 3.8.1 as a dependency in my project but then there is a NoSuchMethodException due to a seeming library mismatch on KafkaUtils.

Help me understand why maven is pulling in the older dependency instead of what is defined in the dependent's pom.xml files.

Solution

  • Spring Boot is a huge project that integrates with tons of other projects and ensuring that all of these different projects are on compatible versions that can run with each other is not a trivial task. To combat this the Spring Boot teams maintain a BOM that specifies version of other projects that they have validated work with each version of Spring Boot. They document every dependency they manage on their site: Managed Coordinates. For many dependencies, versions other than the one specified in the Spring BOM will be compatible but in general it is a best practice to let the Spring BOM manage as many versions as possible. To let the Spring BOM manage a version for you, you can request a group:artifact with the version omitted and the default version in the BOM will be used.

    In your specific case, if we look through Spring Boot's documentation we see that the 3.3.x line of Spring Boot uses the 3.7.x line of Kafka while the 3.4.x line of Spring Boot uses the 3.8.x line of Kafka by default. The best way to get the new version of Kafka would be to move to Spring Boot 3.4.x to ensure it is compatible.