GRPC-Java DEADLINE_EXCEEDED with 5 seconds even if put .withDeadlineAfter(5, TimeUnit.MINUTES)

Question

Not sure if I am missing anything but I have a code block like this

ManagedChannel channel = getManagedChannel(serverId);
var responseObject = ArtifactsGrpc.newBlockingStub(channel).
  .withDeadlineAfter(5, TimeUnit.MINUTES)
  .myGrpcOperation(requestObject);

But I am getting an error

"io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 4.991833115s. [closed=[], open=[[wait_for_ready, buffered_nanos=24207140, remote_addr=backend1/10.0.2.61:6565]]]

The weird part is the 5s (with jitter) in message, een if I set deadline to 5 minutes.

I also moved the new stub creation and with deadline with the grpc operation call as I have read that the deadline starts when withDeadlineAfter is called.

Answer 1

Turns out there was a "timeout" set on the managed channel where the config was

{
  "loadBalancingPolicy": "round_robin",
  "loadBalancingConfig": [
    { "weighted_round_robin": {} },
    { "round_robin": {} },
    { "pick_first": { "shuffleAddressList": true } }
  ],
  "methodConfig": [
    {
      "name": [
        { "service": "" }
      ],
      "waitForReady": true,
      "timeout": "5s", // <-- this one apparently caused the DEADLINE_EXCEEDED
      "retryPolicy": {
        "maxAttempts": 5,
        "initialBackoff": "100ms",
        "backoffMultiplier": 1.0,
        "maxBackoff": "5s",
        "retryableStatusCodes": [
          "UNAVAILABLE"
        ]
      }
    }
  ]
}