Polly CircuitBreaker - Dynamic Duration of Break

Question

What's the correct way to modify the Duration Break on Polly? I know in the documentation they mention implementing (PolicyRegistry). Is there any example of this? I was implementing Polly CircuitBreaker in one WinService.

Answer 1

Circuit Breaker was not designed to use different sleep duration whenever it breaks.

In case of Retry you have the ability to provide a function, called sleepDurationProvider which is called by the policy to determine the actual sleep duration before issuing the next attempt.

So, in short by design it is not supported. I will show you a workaround, but I do not recommend to use it. I will provide a reasoning after the sample code.

---

For the sake of simplicity let's have the following method:

static int Probe()
{
    Console.WriteLine("Probe has been called");
    throw new NotSupportedException();
}

• It prints out the fact that the method has been called then fails immediately.
• IMHO: Not a really useful function :D

Let's define a helper method which increases the time period (>> sleep duration) each and every time whenever it is called:

static IEnumerable<TimeSpan> GetSleepDuration()
{
    for (int i = 1; i < 10; i++)
    {
        yield return TimeSpan.FromSeconds(i);
    }
}

• This replaces (more or less) the Retry's sleepDurationProvider.

Now it's time to define our Circuit Breaker policy:

var sleepDurationProvider = GetSleepDuration().GetEnumerator();
sleepDurationProvider.MoveNext();

var cb = Policy<int>
    .Handle<NotSupportedException>()
    .CircuitBreaker(1, TimeSpan.FromSeconds(0),
    onBreak: (_, __) => {
        Console.WriteLine(sleepDurationProvider.Current.TotalSeconds);
        Thread.Sleep((int)sleepDurationProvider.Current.TotalMilliseconds);
        sleepDurationProvider.MoveNext();
    },
    onReset: () => { },
    onHalfOpen: () => Console.WriteLine("CB half opens"));

• We get an iterator for the GetSleepDuration
• We set the successive failure count to 1 so after each Probe call the CB will break
• We set the durationOfBreak to zero because we will wait inside the onBreak << the workaround
• We print out the sleep duration then we sleep and finally we increase the sleep duration for the next break

In order to ease our testing let's define a retry policy:

var retry = Policy<int>
    .Handle<NotSupportedException>()
    .WaitAndRetryForever(_ => TimeSpan.FromSeconds(0),
    onRetry: (_, __) => Console.WriteLine("Retry is triggered"));

• It handles the Probe's exception
• It retries forever and waits 0 second between each retry call

Let's wire up the policies and run the test:

Policy.Wrap(retry, cb).Execute(() => Probe());

The output will be:

Probe has been called
1
Retry is triggered
CB half opens
Probe has been called
2
Retry is triggered
CB half opens
Probe has been called
3
Retry is triggered
CB half opens
Probe has been called
4
Retry is triggered
CB half opens
Probe has been called
5
Retry is triggered
CB half opens
Probe has been called
6
Retry is triggered
CB half opens
Probe has been called
7
Retry is triggered
CB half opens
Probe has been called
8
Retry is triggered
CB half opens
Probe has been called
9
Retry is triggered
CB half opens
Probe has been called
9
...

1. The Probe has been called and failed
2. CB transits from Open to Broken
3. onBreak has been called, which block the execution for 1 second
4. PolicyWrap escalates the problem to Retry
5. Retry waits 0 second and issues a new attempt
6. CB transits from Broken to Hal-Open
7. The Probe has been called and failed
8. ...

---

The biggest problem with this solution that it is blocking. Unfortunately there is no async version of onBreak. (Retry does have onRetryAsync, in which you could use Task.Delay instead of Thread.Sleep)

The second problem is that it relies on the current implementation. This solution won't work if for example the onBreak is executed on a different thread and the exception is thrown immediately.

---

UPDATE #1

The V8 API has address this concern and you can dynamically define the sleep duration via BreakDurationGenerator.

// The break duration is dynamically determined based on the properties of BreakDurationGeneratorArguments.
var optionsBreakDurationGenerator = new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,
    SamplingDuration = TimeSpan.FromSeconds(10),
    MinimumThroughput = 8,
    BreakDurationGenerator = static args => new ValueTask<TimeSpan>(TimeSpan.FromMinutes(args.FailureCount)),
};