Search code examples
javagradleapache-kafkadependencies

Java gradle kafka-avro-serializer and kafka-schema-registry-client fails to download in the deployment pipeline

Trying to deploy our service in the cloud I am facing the issue that two transitive dependencies could not been downloaded with the error:

Welcome to Gradle 6.7.1!
Here are the highlights of this release:
 - File system watching is ready for production use
 - Declare the version of Java your build requires
 - Java 15 support
For more details see https://docs.gradle.org/6.7.1/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not resolve io.confluent:kafka-avro-serializer:5.3.2.
     Required by:
         project :
         project : > org.apache.beam:beam-runners-google-cloud-dataflow-java:2.32.0 > org.apache.beam:beam-sdks-java-io-kafka:2.32.0
      > Could not resolve io.confluent:kafka-avro-serializer:5.3.2.
         > Could not get resource 'http://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom'.
            > Could not GET 'http://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom'.
               > Connection reset
   > Could not resolve io.confluent:kafka-schema-registry-client:5.3.2.
     Required by:
         project :
         project : > org.apache.beam:beam-runners-google-cloud-dataflow-java:2.32.0 > org.apache.beam:beam-sdks-java-io-kafka:2.32.0
      > Could not resolve io.confluent:kafka-schema-registry-client:5.3.2.
         > Could not get resource 'http://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/5.3.2/kafka-schema-registry-client-5.3.2.pom'.
            > Could not GET 'http://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/5.3.2/kafka-schema-registry-client-5.3.2.pom'.
               > Connection reset

Most resolutions that I could find in the web and here on stackoverflow are centered around adding a new confluent.io gradle repository:

repositories {
    mavenCentral()
    maven {
        url "http://packages.confluent.io/maven/"
    }
}

I tried both http and https. Neither changed anything in the cloud.

I tried to add the transitive dependencies direct to the project gradle. This approach was not succesful too:

    implementation group: 'io.confluent', name: 'kafka-avro-serializer', version: '5.3.2'
    implementation group: 'io.confluent', name: 'kafka-schema-registry-client', version: '5.3.2'

What could be the issue? How to best approach this problem?

Solution

  • The firewall in the cloud was blocking the requests. For some reason they were classified not as http, but as confluent requests, which the firewall did not knew about and therefore were blocking. The aimed at a "confluent" app within the k8s. I don't know where this app comes from, but ... Enabling those requests solved the issue.