I am having intermittent connectivity problems with an old legacy api that sometimes causes a 'BackendConnectionFailure at transfer-response' error to be returned from Azure API Management. From my experience retrying the request to the legacy api is usually successful. I have a retry policy similar to the below that is checking for 5xx status codes, however, the retries do not seem to take place.
<retry
condition="@(context.Response.StatusCode == 500)"
count="10"
interval="10"
max-interval="100"
delta="10"
first-fast-retry="false">
<forward-request buffer-request-body="true" />
</retry>
Upon further research Application Insights seems to indicate that the Backend Dependency has a call status = false, but a Result Code = 200.
Is there any way to detect this condition so that a retry takes place, or any other policies that can be used?
Full policy snippet as suggested:
<policies>
<inbound>
<base />
</inbound>
<backend>
<retry
condition="@(context.Response.StatusCode == 500)"
count="10"
interval="10"
max-interval="100"
delta="10"
first-fast-retry="false">
<forward-request buffer-request-body="true" />
</retry>
</backend>
<outbound>
<set-body>@(context.Response.Body.As<byte[]>())</set-body>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
In your policy above retry covers only receival of response status code and headers from backend. Response body is not proactively read by APIM and instead transferred directly from backend to client piece by piece. That is what "Transfer response" means. By that time all your policies have already completed.
One way to avoid that is to proactively buffer response from backend at APIM side. Try adding as the first thing in outbound:
<set-body>@(context.Response.Body.As<byte[]>())</set-body>