System design - Server sends request but goes down, does it receive response

Question

I have the below basic design for my service.

Flow: User sends a request to browser. The request hits the API service through the load balancer. The load balancer redirects request to available servers and then sends it to external API.

Scenario: Since we have horizontally scaled servers, The request comes to Server 1, server 1 sends this to External API. Before we receive the response from external API, Server 1 goes down. Question:

• In this case, is the response lost (since request came from server 1)?
• How do we handle fail over scenario for this use case?

Answer 1

The request is lost, as the connection created via Server1 is lost.

This is a typical situation to handle in distributed systems. Networks and servers go offline all time, including some temporary blackouts.

Typical approach to handle situations like this is retries - the client waits for some predefined time before declaring the request dead. Then the client may choose to resend the request.

Retry comes with risks. The system may be temporary slow, so retries will overload the system even more. Typically two other strategies are employed: retries have a policy (e.g. retry at most three time, with exponential backoff time) and throttling (systems may explicitly throw away requests and clients should respect that).

Another important concept, one would say the most important, is Idempotency

[copy from wikipedia] Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

The point of idempotent APIs is that they are safe to retry. If a client thinks an operation failed, the client does not know if the request failed before reaching the executor or after the request was fulfilled (like in your example). Idempotent api is safe to retry in any case.