google-cloud-platform grpc google-cloud-dataproc grpc-java

How to get rid of call to CallCredentials2 in grpc api

I'm writing some code for a class project that sends jobs to a dataproc cluster in GCP. I recently ran into an odd error and I'm having trouble wrapping my head around it. The error is as follows:

Exception in thread "Thread-5" java.lang.NoClassDefFoundError: io/grpc/CallCredentials2
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.google.api.gax.grpc.GrpcCallContext.withCredentials(GrpcCallContext.java:160)
at com.google.api.gax.grpc.GrpcCallContext.withCredentials(GrpcCallContext.java:67)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:210)
at com.google.cloud.dataproc.v1.stub.GrpcJobControllerStub.create(GrpcJobControllerStub.java:130)
at com.google.cloud.dataproc.v1.stub.JobControllerStubSettings.createStub(JobControllerStubSettings.java:215)
at com.google.cloud.dataproc.v1.JobControllerClient.<init>(JobControllerClient.java:139)
at com.google.cloud.dataproc.v1.JobControllerClient.create(JobControllerClient.java:120)
at com.shayr.searchEngineGUI.searchEngineGUI.constructInvertedIndices(searchEngineGUI.java:509)
at com.shayr.searchEngineGUI.searchEngineGUI.access$0(searchEngineGUI.java:501)
at com.shayr.searchEngineGUI.searchEngineGUI$2.run(searchEngineGUI.java:474)
Caused by: java.lang.ClassNotFoundException: io.grpc.CallCredentials2
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 22 more

The line in my code that's triggering this issue is:

JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

I tried looking up more about CallCredentials2 initially to figure out what was causing the issue, but I could not find much of anything, other than the class was a temporary workaround while changes were being made to the CallCredentials class and that CallCredentials2 has since been depreciated.

My knowledge of google apis is somewhat limited so it took me a while to figure out how to resolve this. I had to change one of the dependencies for my project so that it changed "grpc-api-1.42.1.jar" to "grpc-api-1.41.0.jar" because the 1.41 version of the api is the one that implemented the CallCredentials2 class.

So my project now works as intended, but I'm still unsure why my code is calling a depreciated class and I'd rather avoid that if at all possible. Does anybody know why this is happening? I'm using very recently updated dependencies so I'm not sure why a depreciated class is being called. Here are the dependencies listed in my pom.xml file before changing the grpc version to 1.41.0:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-datastore</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-dataproc</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>com.google.api-client</groupId>
    <artifactId>google-api-client</artifactId>
    <version>1.32.2</version>
</dependency>

Solution

Using mvn dependency:tree you can discover there's a mix of grpc-java 1.41.0 and 1.42.1 versions in your dependency tree. google-cloud-datastore:2.2.0 brings in grpc-api:1.42.1 but the other dependencies bring in grpc version 1.40.1.

grpc-java recommends always using requireUpperBoundDeps from maven-enforcer to catch Maven silently downgrading dependencies.

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-enforcer-plugin</artifactId>
    <version>1.4.1</version>
    <executions>
      <execution>
        <id>enforce</id>
        <goals>
          <goal>enforce</goal>
        </goals>
        <configuration>
          <rules>
            <requireUpperBoundDeps/>
          </rules>
        </configuration>
      </execution>
    </executions>
  </plugin>

Adding it shows many failures, and they can be fixed by ordering google-cloud-storage after google-cloud-datastore. But fixing the downgrades doesn't resolve your issue.

Let's take a look at the versions as they are now (with the dependency reordering):

$ mvn dependency:tree
[INFO] +- com.google.cloud:google-cloud-datastore:jar:2.2.0:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.42.1:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.42.1:compile
...
[INFO] +- com.google.cloud:google-cloud-dataproc:jar:2.2.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-alts:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.40.1:compile
...

The versions are still broken because io.grpc:grpc-auth 1.40.1 requires exactly io.grpc:grpc-api 1.40.1 but we're using 1.42.1. Maven can detect such cases, but not reliably.

You can fix this by reordering dependencies and explicitly adding some dependencies to force upgrades, but it is tedious. Google Cloud suggests using a BOM (also in docs for datastore, storage) to select versions that are consistent. The <dependencyManagement> section chooses the versions of libraries to use, but doesn't actually depend on them. When using BOMs, you don't need to specify the version of dependencies explicitly that are in the BOM.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>24.0.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-datastore</artifactId>
  </dependency>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
  </dependency>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-dataproc</artifactId>
  </dependency>
  <dependency><!-- not part of the BOM -->
    <groupId>com.google.api-client</groupId>
    <artifactId>google-api-client</artifactId>
    <version>1.32.2</version>
  </dependency>
</dependencies>

And now we can see consistent versions:

$ mvn dependency:tree
...
[INFO] +- com.google.cloud:google-cloud-datastore:jar:2.1.3:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.41.0:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.41.0:compile
...
[INFO] +- com.google.cloud:google-cloud-dataproc:jar:2.2.2:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.41.0:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.41.0:compile
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.41.0:compile
[INFO] |  +- io.grpc:grpc-alts:jar:1.41.0:runtime
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.41.0:runtime
[INFO] |  +- io.grpc:grpc-auth:jar:1.41.0:runtime
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.41.0:runtime
[INFO] |  +- io.grpc:grpc-core:jar:1.41.0:runtime
...