How to add dynamic retry policies using dotnet core and Polly

Question

I have a dotnet core (2.1) Console application and I am using Polly to wrap a segment of my code with a retry policy. This works fine with a simple use case shown below:

private void ProcessRun() 
{
    var policy = Policy.Handle<SocketException>().WaitAndRetryAsync(
                 retryCount: 3
                 sleepDurationProvider: attempt => TimeSpan.FromSeconds(10),
                 onRetry: (exception, calculatedWaitDuration) => 
                 {
                    Console.WriteLine($"Retry policy executed for type SocketException");
                 });

    try
    {
        CancellationTokenSource _cts = new CancellationTokenSource()

        PollyRetryWaitPolicy.ExecuteAsync(async token => {
           MyOperation(token);
        }, _cts.Token)
        .ContinueWith(p => {
           if (p.IsFaulted || p.Status == TaskStatus.Canceled)
           {
                Console.WriteLine("faulted or was cancelled");
           }
        })
        .ConfigureAwait(false);
    }
    catch (Exception ex) {
     Console.WriteLine($"Exception has occurred: {ex.Message}");
    }
}

I then test it using this code:

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new SocketException();
}

When the code is executed, the SocketException is caught as expected.

I was looking a my flexible way to wrap my code with multiple policies instead of one. I changed the code to dynamically add a number of wrapped Polly Retry Policies together, to allow for more than one error type to be caught and easily change the exceptions I'm looking out for. I changed the code to this:

internal PolicyWrap PollyRetryWaitPolicy;

public void AddRetryWaitPolicy<T>(int waitTimeInSeconds, int retryAttempts)
    where T: Exception
{

    // Setup the polly policy that will be added to the executing code.
    var policy = Policy.Handle<T>().WaitAndRetryAsync(
                retryCount: retryAttempts, 
                sleepDurationProvider: attempt => TimeSpan.FromSeconds(waitTimeInSeconds), 
                onRetry: (exception, calculatedWaitDuration) => 
                {
                    Console.WriteLine($"Retry policy executed for type {typeof(T).Name}");
                });

    if (host.PollyRetryWaitPolicy == null)
    {
        // NOTE: Only add this timeout policy as it seems to need at least one
        // policy before it can wrap! (suppose that makes sense).
        var timeoutPolicy = Policy
            .TimeoutAsync(TimeSpan.FromSeconds(waitTimeInSeconds), TimeoutStrategy.Pessimistic);
        PollyRetryWaitPolicy = policy.WrapAsync(timeoutPolicy);
    }
    else
    {
        PollyRetryWaitPolicy.WrapAsync(policy);
    }
}

private void ProcessRun() 
{
    AddRetryWaitPolicy<SocketException>(10, 5);
    AddRetryWaitPolicy<InvalidOperationException>(5, 2);

    try
    {
        Console.WriteLine($"Calling HostedProcess.Run() method. AwaitResult: {awaitResult}");

        CancellationTokenSource _cts = new CancellationTokenSource()

        PollyRetryWaitPolicy.ExecuteAsync(async token => {
           MyOperation(token);
        }, _cts.Token)
        .ContinueWith(p => {
           if (p.IsFaulted || p.Status == TaskStatus.Canceled)
           {
                Console.WriteLine("Process has faulted or was cancelled");
           }
        })
        .ConfigureAwait(false);
    }
    catch (Exception ex) {
     Console.WriteLine($"Exception has occurred: {ex.Message}");
    }
}

When I test with this code, the above code works as expected and is retried 5 times.

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new SocketException();
}

BUT when I try with the following, it does not retry 2 times as expected (it does not retry at all):

private void MyOperation() 
{
    Thread.Sleep(2000);
    throw new InvalidOperationException();
}

What am I doing wrong? The point of all of the above is to dynamically multiple policies as I require. Is there a better way to do with other than WrapPolicy?

Answer 1

Here:

if (host.PollyRetryWaitPolicy == null)
{
    // NOTE: Only add this timeout policy as it seems to need at least one
    // policy before it can wrap! (suppose that makes sense).
    var timeoutPolicy = Policy
        .TimeoutAsync(TimeSpan.FromSeconds(waitTimeInSeconds), TimeoutStrategy.Pessimistic);
    PollyRetryWaitPolicy = policy.WrapAsync(timeoutPolicy);
}
else
{
    PollyRetryWaitPolicy.WrapAsync(policy);
}

it appears the if branch and the else branch take an inconsistent approach to sequencing the retry policy and the timeout policy in the wrap.

• In the if branch, the timeout policy is wrapped inside the retry policy. So the timeout acts as a timeout-per-try.
• In the else branch, any existing retry-and-timeout policy is wrapped outside the new retry policy. So the timeout policy is outside the new retry policy.
  • The new retry policy is set to retry after waitTimeInSeconds; but the timeout policy is also set to timeout the execution after waitTimeInSeconds. So as soon as the first retry (for a second-or-subsequent-configured retry policy) occurs, the timeout aborts the whole execution. So the retry never happens, as you observed.

To cure this you might change the else branch to:

PollyRetryWaitPolicy = policy.WrapAsync(PollyRetryWaitPolicy);

---

Background: See the PolicyWrap wiki recommendations on ordering policies in a PolicyWrap. According to whether you place TimeoutPolicy inside or outside a RetryPolicy, TimeoutPolicy acts (when inside) as timeout-per-try, or (when outside) as overall timeout across all tries.