io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED when querying Secret Manager from App Engine

Question

I'm trying to list secrets in Secret Manager with listSecrets() from an App Engine app, but the grpc requests keeps timing out after 60 s:

com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 59.973305176s.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:51)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
        at io.grpc.internal.DelayedClientCall$CloseListenerRunnable.runInContext(DelayedClientCall.java:406)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed

I verified that I can use other GCP services from App Engine, and that my App Engine default service account has Secret Manager roles. My app is a second generation Java 11 app in the standard environment. I'm using v23.0.0 of libraries-bom (the latest). The same code works fine when I run it locally, and it shouldn't take more than a few seconds, so I'm wondering if it's an issue with the App Engine environment or its service account.

Update: I ended up using google-api-services-secretmanager/google-api-client/google-auth-library-oauth2-http. This snipped worked fine locally and on App Engine.

var projectId = "<TODO>";
var credentials = GoogleCredentials.getApplicationDefault();
var secretManager = new SecretManager.Builder(
        new NetHttpTransport(),
        new GsonFactory(),
        new HttpCredentialsAdapter(credentials))
        .build();

ArrayList<Secret> secrets = new ArrayList<>();
String pageToken = null;
do {
    var response = secretManager.projects().secrets().list("projects/" + projectId)
            .setPageToken(pageToken)
            .execute();
    pageToken = response.getNextPageToken();
    secrets.addAll(response.getSecrets());
} while (pageToken != null);

Answer 1

I finally looked at the logs (not just the exception) and saw this

java.lang.IllegalStateException: Could not find policy 'pick_first'. Make sure its implementation is either registered to LoadBalancerRegistry or included in META-INF/services/io.grpc.LoadBalancerProvider from your jar files.
    at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:92)
    at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:63)
    at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:406)
    at io.grpc.internal.ManagedChannelImpl$RealChannel$2.run(ManagedChannelImpl.java:978)
    at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
    at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
    ...

Which led to a Stackoverflow post and a Github issue.

My build uses the Maven Shade plugin. Two artifacts, io.grpc:grpc-grpclb, and io.grpc:grpc-core have a META-INF/services/io.grpc.LoadBalancerProvider file, but the grpc-core is the one I want. Adding an exclusion to my shade config fixed it:

<filter>
    <artifact>io.grpc:grpc-grpclb</artifact>
    <excludes>
        <exclude>META-INF/services/io.grpc.LoadBalancerProvider</exclude>
    </excludes>
</filter>