Polly: Honoring Retry-After that is communicated with a custom exception?

Question

So I have to use a library that essentially does a POST to a remote system that may choose to throttle the traffic. If it does, it returns 429 and a specific # of seconds to back off in the Retry-After header... at which point the framework reads and parses this value, and essentially does this

throw new ThrottledException(retryAfterSeconds);

How do I set up a Polly policy that will catch this custom exception, and then retry after exception.RetryAfter seconds?

Answer 1

OK, this was a bit more tricky than it needed to be, but only because I was sent on several wild goose chases by inscrutable compiler messages.

In this scenario the retry is communicated by a custom exception of type SigsThrottledException, which has a field that contains the requested backoff time in seconds.

var policy = Policy
    .Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
    .WaitAndRetryAsync(
        retryCount: retries,
        sleepDurationProvider: (i, e, ctx) =>
        {
            var ste = (SigsThrottledException)e;
            return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
        },
        onRetryAsync: async (e, ts, i, ctx) =>
        {
            // Do something here
        };);

This is an example of how to use the policy. You can't just add it to an existing HttpClient or HttpClientFactory. You have to use it explicitly.

[TestMethod]
public async Task SigsPollyRetriesOnThrottle()
{
    var retryResponse = new HttpResponseMessage
    {
        StatusCode = (HttpStatusCode)429,
        Content = new StringContent("{}"),
    };

    retryResponse.Headers.Add("Retry-After", "1");

    var mockMessageHandler = new Mock<HttpMessageHandler>();
    mockMessageHandler.Protected()
        .SetupSequence<Task<HttpResponseMessage>>("SendAsync", ItExpr.IsAny<HttpRequestMessage>(), ItExpr.IsAny<CancellationToken>())
        .ReturnsAsync(retryResponse)
        .ReturnsAsync(new HttpResponseMessage
        {
            StatusCode = HttpStatusCode.OK
        });

    var client = new HttpClient(mockMessageHandler.Object);

    // Retry once after waiting 1 second
    var retryPolicy = Policy
    .Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
    .WaitAndRetryAsync(
        retryCount: 1,
        sleepDurationProvider: (i, e, ctx) =>
        {
            var ste = (SigsThrottledException)e;
            return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
        },
        onRetryAsync: async (e, ts, i, ctx) =>
        {
            // Do something here
        };);

    Stopwatch stopWatch = new Stopwatch();
    stopWatch.Start();
    var response = await retryPolicy.ExecuteAsync(async () =>
    {
        Uri substrateurl = new Uri("https://substrate.office.com/");
        return await SIGSClient.Instance.PostAsync(client, substrateurl, new UserInfo(), "faketoken", new Signal(), Guid.NewGuid()).ConfigureAwait(false);
    }
    );

    Assert.AreEqual(response.StatusCode, HttpStatusCode.OK);
    stopWatch.Stop();
    Assert.IsTrue(stopWatch.ElapsedMilliseconds > 1000); // Make sure we actually waited at least a second
}