I am trying the find out what RPC semantics the gRPC library provides? Is it at-most once? Does it guarantee that an RPC call made by a client is not executed more than once on a server? I couldn't find this explicitly mentioned anywhere in the docs.
From what I understand, gRPC channels have an exponential back-off based retry mechanism of re-initiating TCP connections after transient failures. So, if a server fails after executing an RPC call but before responding, and later comes back up, a client RPC may end up getting executed twice.
Can someone throw more light on this?
gRPC provides at-most-once guarantee by default. Things like retries and similar can change that, but they are opt-in. Another example of this is marking your RPC idempotent or no-side-effects; you've explicitly told gRPC it is okay to replay the RPC.
gRPC may also replay when it is provided assurances elsewhere the RPC was not processed. We call this transparent retries, which we're implementing now, and is based on HTTP/2 semantics where a server tells us the RPC was not seen by the service application.
gRPC's exponential back-off for reconnecting (not retrying RPCs) doesn't change the behavior. Normal RPCs fail outright when the connection is known-bad, but wait-for-ready RPCs that have not been sent are simply delayed until the connection is ready. There is no "replay" of the RPC involved; it is only sent once.