C# Polly Start counting CircuitBreaker Exception only on the last Retry attempt Exception

Question

I'm trying to use Polly on an HttpClient with three policies: Retry, CircuitBreaker, Timeout. The result I would like to achieve is that the CircuitBreaker always starts counting from the last error of the Retry and not from all errors. For example, let's say I have a retry count of 3, I would like behavior like this:

Retry 1 -> HttpRequestException
Retry 2 -> HttpRequestException
Retry 3 -> HttpRequestException
CircuitBreaker Exception count 1
Retry 1 -> HttpRequestException
Retry 2 -> HttpRequestException
Retry 3 -> HttpRequestException
CircuitBreaker Exception count 2
...

This is the class that contains the 3 policies:

public class PollyPoliciesService : IPollyPolicyService, IPollyPolicyConsentsService
{
    private readonly ILogger<PollyPoliciesService> _logger;
    private readonly IDistributedCache _cache;
    private readonly PollySettings _pollySettings;

    public AsyncCircuitBreakerPolicy<HttpResponseMessage> HttpConsentCircuitBreakerPolicy { get; private set; }
    public AsyncRetryPolicy<HttpResponseMessage> HttpConsentsWaitAndLongRetryPolicy { get; private set; }
    public AsyncTimeoutPolicy<HttpResponseMessage> HttpTimeoutPolicy { get; private set; }
    public AsyncRetryPolicy<HttpResponseMessage> HttpRetryPolicy { get; private set; }

    public PollyPoliciesService(ILogger<PollyPoliciesService> logger, IDistributedCache cache,
        IOptions<PollySettings> pollyOptions)
    {
        Guard.Against.Null(logger);
        Guard.Against.Null(cache);
        Guard.Against.Null(pollyOptions);

        _logger = logger;
        _cache = cache;
        _pollySettings = pollyOptions.Value;

        InitializeProperty();
    }

    private void InitializeProperty()
    {
        HttpConsentCircuitBreakerPolicy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .Or<TimeoutRejectedException>()
            .CircuitBreakerAsync(3,TimeSpan.FromSeconds(30), OnBreak, OnReset);

        HttpTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(_pollySettings.HttpTimeoutInSeconds);

        HttpRetryPolicy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .RetryAsync(_pollySettings.HttpRetryNumber);

        HttpConsentsWaitAndLongRetryPolicy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(10, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
    }

    private void OnBreak(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan)
    {
        _logger.LogWarning("CircuitBreaker is open");
    }

    private void OnReset()
    {
        _logger.LogInformation("CircuitBreaker is restored");
    }
}

And this is how I combined the policies::

Policy.WrapAsync(policyService.HttpRetryPolicy,policyConsentsService.HttpConsentCircuitBreakerPolicy,policyService.HttpTimeoutPolicy);

Unfortunately what I get is that after the third exception raised by the Retry the CircuitBreaker is triggered. How can I solve the problem?
Thanks

Answer 1

Unfortunately what I get is that after the third exception raised by the Retry the CircuitBreaker is triggered.

Since the retry is the inner policy and the circuit breaker is the outer policy that's why after the 3rd retry attempt the cb does not break. BUT it does throw the HttpRequestException.

In the documentation you can find a flow diagram.
I've highlight which path is executed in this particular case:

I've simplified your code to show that the CB does not break after the first retry iteration.

static async Task Main(string[] args)
{
    var cb = Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .CircuitBreakerAsync(3, TimeSpan.FromSeconds(1));

    var retry = Policy<HttpResponseMessage>
        .Handle<HttpRequestException>()
        .RetryAsync(3);

    var combined = Policy.WrapAsync(cb, retry);

    for (int i = 0; i < 3; i++)
    {
        Console.WriteLine(i+1);
        try
        {
            await combined.ExecuteAsync(Get);
        }
        catch (HttpRequestException)
        {
            Console.WriteLine(cb.CircuitState);
        }
    }           
}

static Task<HttpResponseMessage> Get()
{
    Console.WriteLine("Called");
    throw new HttpRequestException();
}

Then the output will be

1
Called
Called
Called
Called
Closed
2
Called
Called
Called
Called
Closed
3
Called
Called
Called
Called
Open

As you can see the CB opened after the third iteration.

• If we would have a 4th iteration then that would result in a BrokenCircuitException