why error message "Operation failed after 3 times" appears even there is no re-try for error code 100

I want to re-try execution of code block if error code = 100 and if all re-try fails I want to print the message "Operation failed after 3 times" for this case only.

I have below Polly policy defined,

var retryPolicy = Policy
            .Handle<Exception>(ex =>
            {
                var errorMessage = ex.InnerException?.Message;
                if (errorMessage == null) return false;
                try
                {
                    dynamic error = JsonConvert.DeserializeObject(errorMessage);
                    if (error != null && error.errorCode == 100)
                    {
                        Console.WriteLine("Try to run again...");
                        return true;
                    }

                    Console.WriteLine("Error occurred: " + ex.Message);
                    return false;
                }
                catch (Exception)
                {
                    Console.WriteLine("Exception Error occurred: " + ex.Message);
                    return false;
                }
            })
            .WaitAndRetryAsync(2, _ => TimeSpan.FromSeconds(10));

And below is the code for policy execution,

try
{
    var myClass = new MyClass();
    await retryPolicy.ExecuteAsync(async () => await myClass.Get());
}
catch (Exception)
{
    Console.WriteLine("Operation failed after 3 times");
}

For below code everything perfect and I am getting desired result as well,

Try to run again...
Try to run again...
Try to run again...
Operation failed after 3 times

public async Task Get()
{
    await Task.Delay(1);
    throw new Exception("message", new Exception("{\"message\":\"inner exception\",\"errorCode\":\"100\"}"));
}

But when I am execution below code ( no error code = 100), then my re-try not happening but the message "Operation failed after 3 times" is also printing in console. What's the reason for it and how to avoid it?

Exception Error occurred: message
Operation failed after 3 times

public async Task Get()
{
    await Task.Delay(1);
    throw new Exception("message", new Exception("hey you"));
}

Solution

Let me present here a solution which shows how should you solve this problem with Polly.

Retries vs Attempts

Let's start with clarifying some terminology.

In your code sample you have set the retryCount to 2 but you have 3 Try to run again... messages on your Console. The reason behind this is that in total you had 3 attempts: 1 initial attempt and 2 retries.

Because you have put your logging inside the exceptionPredicate that's why it is evaluated three times:

After the initial attempt
After the first retry attempt
After the second retry attempt

The last one is bit odd since it does not trigger a retry. Why? Because you would exceed the retry count with that.

Later in this post we will discuss where should you put the logging.

The `exceptionPredicate`

Please try to keep this predicate as simple as possible. As you have seen it it is evaluated after each attempt (not after each retry)!

Rather than having a try-catch inside this predicate you can instruct the Json.Net deserializer to silently ignore errors if it can not parse the input as json.

var silentlyIgnoreError = new JsonSerializerSettings 
{ 
    Error = (_, args) => args.ErrorContext.Handled = true 
};

With this settings your predicate could be streamlined like this

.Handle<Exception>(ex =>
{
    var errorMessage = ex.InnerException?.Message;
    if (errorMessage == null) 
       return false;
    if (JsonConvert.DeserializeObject(errorMessage, silentlyIgnoreError) == null) 
       return false;

    var errorCode = (string?)JObject.Parse(errorMessage)["errorCode"];
    return int.TryParse(errorCode, out int errorNumber) && errorNumber == 100;
})

If the exception does not contain an inner then do not retry
If the inner excepion's message can't be parsed as json then do not retry
If the json does not contain an errorCode field then do not retry
If the json contains an errorCode field but the value is not an integer then do not retry
If the json contains an errorCode field with an integer value but it's different than 100 then do not retry
Otherwise do retry :D

As you can see there is no logging here.

Logging

The logging logic should be placed inside the onRetry/onRetryAsync delegate which is executed when the policy has already decided that is should be triggered but before the sleep.

.WaitAndRetryAsync(2,
    _ => TimeSpan.FromSeconds(10),
    (ex, _, ctx) =>
    {
        Console.WriteLine($"Try to run again... ({ex.Message})");
        ctx.IncreaseRetryCount();
    });

With this setup you would see only two Try to run again... lines
- One after the initial attempt
- One after the first retry attempt
I've used a special overload of the onRetry which has access to the Context
- It gives us ability to store information between retry attempts
- It also allows us to access that information after the policy execution

The usage of `Context`

I've defined two extension methods to ease the usage of the Context, which is a Dictionary<string, object> under the hood

public static class ContextExtensions
{
    private static readonly string key = "RetryCount";

    public static void IncreaseRetryCount(this Context context)
    {
        var retryCount = GetRetryCount(context);
        context[key] = ++retryCount;    
    }

    public static int GetRetryCount(this Context context)
    {
        context.TryGetValue(key, out object count);
        return count != null ? (int)count : 0;
    }
}

The IncreaseRetryCount is called whenever a retry will be triggered
The GetRetryCount is called after the execution of the policy

The execution of the policy

You can execute the policy not just with the Execute/ExecuteAsync but with the ExecuteAndCapture/ExecuteAndCaptureAsync as well.

It returns a PolicyResult/PolicyResult<T> object which has the following properties:

Outcome: Whether the policy/chain of policies succeeded or failed
FinalException: In case of failure the final exception
Context: That Context object which was used during the execution
(Result: If the policy had been defined in a way that it should return something)

As you might expect in case of a Failure it won't throw an exception.

If you would use ExecuteAndCaptureAsync then your code would look like this:

var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempt(s)");

For the sake of completeness here is the full source code

var silentlyIgnoreError = new JsonSerializerSettings { Error = (_, args) => args.ErrorContext.Handled = true };
var retryPolicy = Policy
    .Handle<Exception>(ex =>
    {
        var errorMessage = ex.InnerException?.Message;
        if (errorMessage == null) return false;
        if (JsonConvert.DeserializeObject(errorMessage, silentlyIgnoreError) == null) return false;

        var errorCode = (string?)JObject.Parse(errorMessage)["errorCode"];
        return int.TryParse(errorCode, out int errorNumber) && errorNumber == 100;
    })
    .WaitAndRetryAsync(2,
        _ => TimeSpan.FromSeconds(10),
        (ex, _, ctx) =>
        {
            Console.WriteLine($"Try to run again... ({ex.Message})");
            ctx.IncreaseRetryCount();
        });

var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempts");

public static class ContextExtensions
{
    private static readonly string key = "RetryCount";

    public static void IncreaseRetryCount(this Context context)
    {
        var retryCount = GetRetryCount(context);
        context[key] = ++retryCount;
    }

    public static int GetRetryCount(this Context context)
    {
        context.TryGetValue(key, out object count);
        return count != null ? (int)count : 0;
    }
}

UPDATE #1: Some correction

This piece of code:

var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempts");

assumed that the operation would always fail.

The correct way of handling this should be written like this:

var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
var outcome = result.Outcome == OutcomeType.Successful ? "completed" : "failed";
Console.WriteLine($"Operation has {outcome} after the initial attempt + {result.Context.GetRetryCount()} retry attempts");

The OutcomeType can be either Successful or Failure.