Azure API management not sending retry when function app disabled

Question

I've configured an API Management policy to retry to a second region if the first encounters an error.

However, when I disable the primary function app, based in Sydney, there are no retries recorded in Application insights - only the original failing request. There are no function calls recorded in the Melbourne function app.

I've tried a bunch of different configurations. Both region function apps are running fine.

I've got the copied the code below, is there something that I'm missing or not understanding?

Thanks in advance

API Policy

<policies>
  <inbound>
    <base />
    <choose>
      <when condition="@(context.Variables.GetValueOrDefault<int>('RetryCounter', 0) == 0)">
        <set-backend-service base-url="{{syndey-function-app}}" />
      </when>
      <otherwise>
        <set-backend-service base-url="{{melbourne-function-app}}" />
      </otherwise>
    </choose>
  </inbound>
  <backend>
    <retry count="1" interval="0" first-fast-retry="true" condition="@(
        context.Response.StatusCode > 400
      )">
      <set-variable name="RetryCounter" value="@(context.Variables.GetValueOrDefault<int>('RetryCounter', 0) + 1)" />
      <forward-request buffer-request-body="true" timeout="15"/>
    </retry>
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <base />
  </on-error>
</policies>

403 Error Response

<h1 id="unavailable">Error 403 - This web app is stopped.</h1>
<p id="tryAgain">
The web app you have attempted to reach is currently stopped and does not 
accept any requests. Please try to reload the page or visit it again soon.
</p>

<p id="toAdmin">
If you are the web app administrator, please find the common 403 error 
scenarios and resolution <a href="https://go.microsoft.com/fwlink/?linkid=2095007" 
target="_blank">here</a>. For further troubleshooting tools and recommendations, 
please visit <a href="https://portal.azure.com/">Azure Portal</a>.
</p>

Answer 1

I had to replace the single quotes to double quotes because of an error on save:

context.Variables.GetValueOrDefault<int>('RetryCounter', 0) => context.Variables.GetValueOrDefault<int>("RetryCounter", 0)

For testing I created two mocked services:

Returns status code 200:
https://rfqapiservicey27itmeb4cf7q.azure-api.net/echo/200/test

Returns status code 403:
https://rfqapiservicey27itmeb4cf7q.azure-api.net/echo/403/test

The retry section in the backend section will never return to the inbound section. So the choose condition was only hit at the very beginning.

Therefore, the choose condition has to be moved to the backend section.

<policies>
    <inbound>
        <base />
        <set-backend-service base-url="https://rfqapiservicey27itmeb4cf7q.azure-api.net/echo/403" />
        <!--
            <set-backend-service base-url="{{syndey-function-app}}" />
         -->
    </inbound>
    <backend>
        <retry count="1" interval="0" first-fast-retry="true" condition="@(
        context.Response.StatusCode > 400
      )">
            <set-variable name="RetryCounter" value="@(context.Variables.GetValueOrDefault<int>("RetryCounter", 0) + 1)" />
            <choose>
                <when condition="@(context.Variables.GetValueOrDefault<int>("RetryCounter", 0) > 1)">
                    <set-backend-service base-url="https://rfqapiservicey27itmeb4cf7q.azure-api.net/echo/200" />
                    <!--
                        <set-backend-service base-url="{{melbourne-function-app}}" />
                    -->
                </when>
            </choose>
            <forward-request buffer-request-body="true" timeout="15" />
        </retry>
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

My sample operation executes as expected:

https://rfqapiservicey27itmeb4cf7q.azure-api.net/sample/test

The first hit in the backend requests the 403 resource:

This request is failing, and the retry policy sets the 200 in the backend section:

This second request is successful when will return 200 and leave the backend section: