Search code examples
javascriptnode.jsgrpcgrpc-node

nodejs grpc-js reconnection after server restart


I am experiencing some strange behavior with @grpc/grpc-js client reconnection

Once the client connected successfully and the server restart, the client keep looping in connecting state

How should I handle reconnecting when server drop ?

Thanks a lot for your help

Client debug:

D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | load_balancing_call | [30] Pick called
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | load_balancing_call | [30] Pick result: QUEUE subchannel: null status: undefined undefined
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.ref | configSelectionQueue.length=0 pickQueue.length=1
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | connectivity_state | (1) dns:10.10.10.10:50051 CONNECTING -> IDLE
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | resolving_load_balancer | dns:10.10.10.10:50051 IDLE -> CONNECTING
D 2023-11-16T17:23:49.137Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2023-11-16T17:23:49.138Z | v1.9.10 67824 | load_balancing_call | [30] Pick called
D 2023-11-16T17:23:49.138Z | v1.9.10 67824 | load_balancing_call | [30] Pick result: QUEUE subchannel: null status: undefined undefined
D 2023-11-16T17:23:49.138Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.ref | configSelectionQueue.length=0 pickQueue.length=1
D 2023-11-16T17:23:49.138Z | v1.9.10 67824 | connectivity_state | (1) dns:10.10.10.10:50051 IDLE -> CONNECTING
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | resolving_load_balancer | dns:10.10.10.10:50051 CONNECTING -> IDLE
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | load_balancing_call | [30] Pick called
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | load_balancing_call | [30] Pick result: QUEUE subchannel: null status: undefined undefined
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.ref | configSelectionQueue.length=0 pickQueue.length=1
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | connectivity_state | (1) dns:10.10.10.10:50051 CONNECTING -> IDLE
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | resolving_load_balancer | dns:10.10.10.10:50051 IDLE -> CONNECTING
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | load_balancing_call | [30] Pick called
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | load_balancing_call | [30] Pick result: QUEUE subchannel: null status: undefined undefined
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | channel | (1) dns:10.10.10.10:50051 callRefTimer.ref | configSelectionQueue.length=0 pickQueue.length=1
D 2023-11-16T17:23:50.140Z | v1.9.10 67824 | connectivity_state | (1) dns:10.10.10.10:50051 IDLE -> CONNECTING

On my server side, I receive absolutely nothing. I used tcpdump and there's no traffic

Here's my client settings:

const client = new Service(url, credentials.createInsecure(), {
          'grpc.max_reconnect_backoff_ms': 1000,
          'grpc.initial_reconnect_backoff_ms': 5000,
          'grpc.keepalive_time_ms': 1000,
          'grpc.keepalive_permit_without_calls': 1,
          'grpc.keepalive_timeout_ms': 500,
          'grpc.max_send_message_length': 1024 * 1024 * 25,
          'grpc.max_receive_message_length': 1024 * 1024 * 25,
        });

Solution

  • That is a bug in the grpc-js library. It has been fixed in version 1.9.11.