Search code examples
javamavengoogle-cloud-platformgoogle-cloud-pubsubmaven-assembly-plugin

PubSub Runtime Error, Could not find policy 'pick_first' in META-INF/services/io.grpc.LoadBalancerProvider

I have a maven project with pubsub and I use maven-assembly-plugin to generate a jar file with dependencies.

With pubsub version 1.104.1, the jar file is working properly and I tried to update it to latest version 1.123.13. After updating, my application works as expected with exec-maven-plugin, but the same code is not working with a maven-assembly-plugin-generated jar file.

POM file and Error as follows.

<!-- language: xml -->
    <?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>org.example</groupId>
        <artifactId>test</artifactId>
        <version>1.0-SNAPSHOT</version>
    
        <properties>
            <java.version>8</java.version>
            <junit-version>4.4</junit-version>
        </properties>
    
        <dependencies>
           
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>google-cloud-pubsub</artifactId>
                <version>1.123.13</version>
            </dependency>
    
        </dependencies>
    
        <build>
    
            <extensions>
                <extension>
                    <groupId>kr.motd.maven</groupId>
                    <artifactId>os-maven-plugin</artifactId>
                    <version>1.5.0.Final</version>
                </extension>
            </extensions>
    
            <plugins>
    
                <plugin>
                    <groupId>org.jacoco</groupId>
                    <artifactId>jacoco-maven-plugin</artifactId>
                    <version>0.8.5</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>prepare-agent</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>report</id>
                            <phase>prepare-package</phase>
                            <goals>
                                <goal>report</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>jacoco-check</id>
                            <goals>
                                <goal>check</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
    
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <version>3.6.0</version>
                    <configuration>
                        <archive>
                            <manifest>
                                <addClasspath>true</addClasspath>
                                <mainClass>Main</mainClass>
                            </manifest>
                        </archive>
                        <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                        </descriptorRefs>
                    </configuration>
                    <executions>
                        <execution>
                            <id>assemble-all</id>
                            <phase>package</phase>
                            <goals>
                                <goal>single</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
    
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.7.0</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
            </plugins>
    
            <pluginManagement>
                <plugins>
                    <plugin>
                        <groupId>org.codehaus.mojo</groupId>
                        <artifactId>exec-maven-plugin</artifactId>
                        <version>1.6.0</version>
                        <configuration>
                            <cleanupDaemonThreads>false</cleanupDaemonThreads>
                        </configuration>
                    </plugin>
                    <plugin>
                        <groupId>org.sonarsource.scanner.maven</groupId>
                        <artifactId>sonar-maven-plugin</artifactId>
                        <version>3.7.0.1746</version>
                    </plugin>
                </plugins>
            </pluginManagement>
    
        </build>
    
    </project>

Error Log

Jun 13, 2023 12:15:11 PM io.grpc.internal.ManagedChannelImpl$2 uncaughtException
        SEVERE: [Channel<3>: (pubsub.googleapis.com:443)] Uncaught exception in the SynchronizationContext. Panic!
        java.lang.IllegalStateException: Could not find policy 'pick_first'. Make sure its implementation is either registered to LoadBalancerRegistry or included in META-INF/services/io.grpc.LoadBalancerProvider from your jar files.
                at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:94)
                at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:61)
                at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:413)
                at io.grpc.internal.ManagedChannelImpl$RealChannel$2.run(ManagedChannelImpl.java:968)
                at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
                at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
                at io.grpc.internal.ManagedChannelImpl$RealChannel.newCall(ManagedChannelImpl.java:965)
                at com.google.api.gax.grpc.GrpcChannelUUIDInterceptor.interceptCall(GrpcChannelUUIDInterceptor.java:52)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:80)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:54)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:907)
                at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
                at com.google.api.gax.grpc.ChannelPool$AffinityChannel.newCall(ChannelPool.java:516)
                at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:145)
                at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:99)
                at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:60)
                at com.google.api.gax.grpc.GrpcUnaryRequestParamCallable.futureCall(GrpcUnaryRequestParamCallable.java:65)
                at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
                at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:87)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
                at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
                at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
                at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:319)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:224)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.createTopic(PubSubConsumer.java:232)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.startConsume(PubSubConsumer.java:197)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.run(PubSubConsumer.java:138)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at java.base/java.lang.Thread.run(Thread.java:830)
        
        Exception in thread "pool-1-thread-1" com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
                at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:110)
                at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
                at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
                at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
                at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
                at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
                at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
                at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
                at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
                at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:761)
                at com.google.common.util.concurrent.ForwardingListenableFuture.addListener(ForwardingListenableFuture.java:47)
                at com.google.api.core.ApiFutureToListenableFuture.addListener(ApiFutureToListenableFuture.java:49)
                at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1105)
                at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:79)
                at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:67)
                at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:87)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
                at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
                at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
                at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:319)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:224)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.createTopic(PubSubConsumer.java:232)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.startConsume(PubSubConsumer.java:197)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.run(PubSubConsumer.java:138)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at java.base/java.lang.Thread.run(Thread.java:830)
                Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                        ... 9 more
        Caused by: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
                at io.grpc.Status.asRuntimeException(Status.java:539)
                at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
                at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
                at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
                at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
                at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541)
                at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
                at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
                at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
                at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
                ... 3 more
        Caused by: java.lang.IllegalStateException: Could not find policy 'pick_first'. Make sure its implementation is either registered to LoadBalancerRegistry or included in META-INF/services/io.grpc.LoadBalancerProvider from your jar files.
                at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:94)
                at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:61)
                at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:413)
                at io.grpc.internal.ManagedChannelImpl$RealChannel$2.run(ManagedChannelImpl.java:968)
                at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
                at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
                at io.grpc.internal.ManagedChannelImpl$RealChannel.newCall(ManagedChannelImpl.java:965)
                at com.google.api.gax.grpc.GrpcChannelUUIDInterceptor.interceptCall(GrpcChannelUUIDInterceptor.java:52)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:80)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:54)
                at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
                at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:907)
                at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
                at com.google.api.gax.grpc.ChannelPool$AffinityChannel.newCall(ChannelPool.java:516)
                at com.google.api.gax.grpc.ChannelPool.newCall(ChannelPool.java:145)
                at com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:99)
                at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:60)
                at com.google.api.gax.grpc.GrpcUnaryRequestParamCallable.futureCall(GrpcUnaryRequestParamCallable.java:65)
                at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
                at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:87)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
                at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
                at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
                at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
                at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:319)
                at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:224)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.createTopic(PubSubConsumer.java:232)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.startConsume(PubSubConsumer.java:197)
                at com.stl.sciocardio.common.pubsub.PubSubConsumer.run(PubSubConsumer.java:138)
                ... 3 more

What is the reason for this?

Solution

  • Issue getting when same file provide by many dependencies. At jar file generation time it select one file. If required details not available in selected file, this kind of issues can happen.

    Following is the solution for my issue.

    1. Replace maven-assembly-plugin from maven-shade-plugin[1]. This plugin create executable jar file. At the build time it shows overlapping resources.

    2. At build result shows META-INF/services/io.grpc.LoadBalancerProvider overlapping resource.

      [WARNING] grpc-core-1.55.1.jar, grpc-grpclb-1.55.1.jar, grpc-rls-1.55.1.jar, grpc-services-1.55.1.jar, grpc-xds-1.55.1.jar define 1 overlapping resource: [WARNING] - META-INF/services/io.grpc.LoadBalancerProvider

    3. Here it shows three dependencies have io.grpc.LoadBalancerProvider file. But their contents are different. Only grpc-core has pick_first record. If another file select at jar file generation, run time pick_first record not found.

    4. With shade plugin there is a option to merge all same name service files. Following transformer need to add in pom.xml [2] <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>

    Shade plugin details in pom.xml

        <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.4.1</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>Main Class</mainClass>
                        </transformer>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                    </transformers>
                </configuration>
            </execution>
        </executions>
    </plugin>

    [1]. https://maven.apache.org/plugins/maven-shade-plugin/examples/executable-jar.html

    [2]. https://maven.apache.org/plugins/maven-shade-plugin/apidocs/org/apache/maven/plugins/shade/resource/ServicesResourceTransformer.html