The gRPC Protocol Spec specifies that the grpc-timeout header uses relative values (eg: 100m). However, gRPC prefers absolute deadlines and I'm struggling to understand how this absolute deadline is communicated over the wire.
for eg: Lets say that the one-way geographic latency between the client and server is 100ms. Let's use a linear scale for time in millis for simplicity.
If a client makes a requests at time 0 with "grpc-timeout: 300m" Header then the server will receive the request at time 100. How does the server recognize that it only has 200ms remaining to process the request?
How does the server recognize that it only has 200ms remaining to process the request?
It doesn't. Network latency is not taken into account. And understand in your example that the server actually only has 100ms to process, because it will take 100ms for the response to be received. The round-trip time (RTT) is 200ms. Note that RTTs are dynamic (they change over time) and the network links may not be symmetric (it might be 80ms in one direction and 120ms in the other).
There's no guarantee the client and server have their clocks synchronized. So there is no "absolute" time for gRPC to rely on when communicating between them. There's also no guarantee that the clocks progress at the same rate; one clock may be slower than the other (aka clock drift). Using a timeout on-the-wire generally covers up any issues caused by clock drift.
Communicating a timeout on-the-wire but deadlines within-a-process is simple and effective. Trying to be 100% precise about the deadline requires supporting assumptions or systems. In practice, the RTT shouldn't be a substantial portion of the time remaining before the deadline as that does not provide enough time for lost packets and retransmits (ignoring that gRPC might even need to establish a TCP connection). If the deadline is aggressive compared to the RTT, then the server computing results that went unused would just need to be accepted or requires vastly different networking designs and tradeoffs.