Search code examples
c#.netdockerpollycircuit-breaker

Can't execute circuit breaker after containerization


I have a .NET 5 application, divided in microservices. I implemented the Circuit Breaker pattern using the Microsoft library and Polly.

Everything was tested and working accordingly - locally. But when I try to run with docker, if none of the microservices is down, it works perfectly, all the requests have responses (the expected ones).

On the contrary, when I put one of the microservices down and try to test and see if the circuit is open (getting the response with that information) it simply times out, returning an exception in which a task was not completed (timeout).

How can I solve this? I'm using the http port to run docker, I have tried to disable the httpsRedirection in Startup.cs as well as making the requests either with http and https, but none of these latter was successful. I am really out of ideas here. Below is an example of a microservice(with the relevant code of the question) and the respective docker file:

OrchAuth.cs:

services.AddControllers();
services.AddHttpClient<ISearchCommunicationServiceWatchables, SearchRESTCommunicationServiceWatchables>("Watchables")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<IUsersCommunicationService, UsersRESTCommunicationService>("Users")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<IUserPreferencesService, UserPreferencesService>("UserPreferences")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
services.AddHttpClient<ISearchCommunicationServiceBooks, SearchRESTComunicationServiceBooks>("Books")
     .SetHandlerLifetime(TimeSpan.FromMinutes(1))
     .AddPolicyHandler(GetRetryPolicy())
     .AddPolicyHandler(GetCircuitBreakerPolicy());
...
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    Random jitterer = new ();
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(2, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))  // exponential back-off: 2, 4, 8 etc
            + TimeSpan.FromMilliseconds(jitterer.Next(0, 1000))); // plus some jitter: up to 1 second);
}

static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10));
}

The docker file:

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["boomoseries-OrchAuth-api/boomoseries-OrchAuth-api.csproj", "boomoseries-OrchAuth-api/"]
RUN dotnet restore "boomoseries-OrchAuth-api/boomoseries-OrchAuth-api.csproj"
COPY . .
WORKDIR "/src/boomoseries-OrchAuth-api"
RUN dotnet build "boomoseries-OrchAuth-api.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "boomoseries-OrchAuth-api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENV USERS_HOST=http://host.docker.internal:5020/Users
ENV PREFS_HOST=http://host.docker.internal:5024/UserPreferences/Favorites
ENV SEARCH_HOST=http://host.docker.internal:5018/api/v1/Search
ENTRYPOINT ["dotnet", "boomoseries-OrchAuth-api.dll"]

Solution

  • The problem has been solved. The issue was the following: the circuit breaker policy is configured to handle HTTP errors of type 5XX and 408, and I was not getting any of these as a response (when running docker): I was getting "blank". This does not happen locally because of the host actively refusing connection, being able to trigger and open the circuit. So for this, I had to configure the timeout policy in Polly, throw the exception and let the circuit breaker policy handle it (code below).

     public static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
        {
            return HttpPolicyExtensions
                .HandleTransientHttpError()
                .Or<TimeoutRejectedException>()
                .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10));
        }
    

    Another detail was the order in which this was configured in the startup: the circuit breaker policy had to be implemented BEFORE the retry policy (code below).

    public static IHttpClientBuilder ConfigHttpClient<TInterface, TClass>(this IServiceCollection services, string httpClientName)
            where TInterface : class
            where TClass : class, TInterface
        {
    
           return services.AddHttpClient<TInterface, TClass>(httpClientName)
               .AddPolicyHandler(Startup.GetCircuitBreakerPolicy())
               .AddPolicyHandler(Startup.GetRetryPolicy())
               .AddPolicyHandler(Policy.TimeoutAsync<HttpResponseMessage>(3));
    
        }