How to deal with 100 seconds timeouts while using Poly retry policy

Question

I'm using retry policy in .net core application and am getting timeouts after exceeding 100 seconds period. Might I use Poly in some incorrect way or it's by design and only timeout period increase might help?

Here is the way I use Poly: Startup:

// populate timeouts array from appsettings
var resilencyOptions = services.BuildServiceProvider().GetRequiredService<IOptions<ResiliencyOptions>>().Value;
var attempts = resilencyOptions.TimeOutsInSeconds.Count;
TimeSpan[] timeouts = new TimeSpan[attempts];
int i = 0;

foreach (var timeout in resilencyOptions.TimeOutsInSeconds)
{
    timeouts[i++] = TimeSpan.FromSeconds(timeout);
}

// register
services.AddTransient<LoggingDelegatingHandler>();
services.AddHttpClient<IMyClient, MyClient>()
    .AddHttpMessageHandler<LoggingDelegatingHandler>()
    .AddPolicyHandler(ResiliencyPolicy.GetRetryPolicy(attempts, timeouts))
    .AddPolicyHandler(ResiliencyPolicy.GetCircuitBreakerPolicy());

Library:

/// <summary>
/// Resiliency policy.
/// </summary>
public class ResiliencyPolicy
{
    /// <summary>
    /// Get a retry policy.
    /// </summary>
    /// <param name="numberofAttempts"> Количество попыток.</param>
    /// <param name="timeOfAttempts"> Массив с таймаутами между попытками, если передается неполный или пустой, попытки делаются в секундах 2^.</param>
    /// <returns></returns>
    public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(int numberofAttempts = 5, TimeSpan[] timeOfAttempts = null)
    {
        //  In case timeOfAttempts is null or its elements count doesnt correspond to number of attempts provided,
        //  we will wait for:
        //  2 ^ 1 = 2 seconds then
        //  2 ^ 2 = 4 seconds then
        //  2 ^ 3 = 8 seconds then
        //  2 ^ 4 = 16 seconds then
        //  2 ^ 5 = 32 seconds

        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(
                retryCount: numberofAttempts,
                sleepDurationProvider: retryAttempt =>  ((timeOfAttempts == null) || (timeOfAttempts.Length != numberofAttempts)) ?
                                                        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) :
                                                        timeOfAttempts[retryAttempt],
                onRetry: (exception, retryCount, context) =>
                {
                    Logging.Global.LogError($"Retry {retryCount} of {context.PolicyKey} at {context.OperationKey}, due to: {exception}.");
                });
    }

    /// <summary>
    /// Get circuit breaker policy.
    /// </summary>
    /// <param name="numberofAttempts">количество попыток</param>
    /// <param name="durationOfBreaksInSeconds">количество секунд (таймаут) между попытками</param>
    /// <returns></returns>
    public static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy(int numberofAttempts = 5, int durationOfBreaksInSeconds = 30)
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(
                handledEventsAllowedBeforeBreaking: numberofAttempts,
                durationOfBreak: TimeSpan.FromSeconds(durationOfBreaksInSeconds)
            );
    }
}

Calling from custom http client:

public class MyClient : IMyClient
{
    private readonly HttpClient _httpClient;
    private readonly ILogger<MyClient> _logger;

    public MyClient(HttpClient httpClient, ILogger<MyClient> logger)
    {
        _httpClient = httpClient;
        _logger = logger;
    }

    public async Task<bool> Notify(string url, Guid id, string orderId, int state, int category, DateTime date, CancellationToken cancellationToken)
    {
        // prepare request
        var request = new
        {
            Id = id,
            OrderId = orderId,
            State = state,
            Category = category,
            Date = date
        };

        var data = new StringContent(JsonSerializer.Serialize(request), Encoding.UTF8, "application/json");

        // send request
        _logger.LogInformation("sending request to {url}", url);
        var response = await _httpClient.PostAsync(url, data, cancellationToken);

        // process response
        if (response.IsSuccessStatusCode)
            return true;

        var content = await response.Content.ReadAsStringAsync(cancellationToken);

        response.Content?.Dispose();

        throw new HttpRequestException($"{response.ReasonPhrase}. {content.Replace("\"", "").TrimEnd()}", null, response.StatusCode);
    }
}

Controller simulating endpoint availability:

[ApiController]
[Route("[controller]")]
public class RabbitController : ControllerBase
{
    private static int _numAttempts;

    public RabbitController(IBus client)
    {
        _client = client;
    }

    [HttpPost("ProcessTestREST")]
    public IActionResult ProcessTestREST(Object data)
    {
        _numAttempts++;
        if (_numAttempts%4==3)
        {
            return Ok();
        }
        else
        {
            return StatusCode((int)HttpStatusCode.InternalServerError, "Something went wrong");
        }
    }
}    

I am getting this error:

"The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing."

Answer 1

Check https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0#dynamically-select-policies

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(10));
var longTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(30));

builder.Services.AddHttpClient("PollyDynamic")
    .AddPolicyHandler(httpRequestMessage =>
        httpRequestMessage.Method == HttpMethod.Get ? timeoutPolicy : longTimeoutPolicy);

The timeout policy should be set during the AddHttpClient phase to override the 100 seconds default value as defined in the official documentation.

Your timeout for polly related requests, should cover the biggest value of your retry policy.

Beware to use custom clients in case you want to ignore the retries, so that the timeout is the default one.