Unit test Polly - Check whether the retry policy is triggered on timeout / error

Question

Problem stmt : I have a service which fetches result from graphql sometime due to server issue the service might throw 500 error

Solution: To resolve the above issue I needed to write a retry logic to retry the service when timeout occurs.

Obstacle : I don't know how to assert the whether the given logic is calling the service three times as specified. Any help is appreciated.

I created a retry policy to retry if the given client is timeout after some time.

public override void ConfigureServices(IServiceCollection services)
{
  services.AddHttpClient<GraphQueryService>(Constants.PPQClient)
        .ConfigureHttpClient(client =>
        {
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            client.Timeout = TimeSpan.FromMilliseconds(Constants.ClientTimeOut);
        }).AddRetryPolicy();
}

RetryLogic :

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Polly;

namespace PA.Com.Integration
{
    public static class HttpClientBuilderExtensions
    {
        public static IHttpClientBuilder AddRetryPolicy(this IHttpClientBuilder builder)
        {
            var serviceProvider = builder.Services.BuildServiceProvider();

            var options = serviceProvider.GetRequiredService<IOptions<RetryOptions>>();

            return builder.AddTransientHttpErrorPolicy(b => b.WaitAndRetryAsync(new[]
            {
                options.Value.RetryDelay1,
                options.Value.RetryDelay2,
                options.Value.RetryDelay3
            }));
        }
    }
}

I'm new to unit testing I believe I called the code to check the timeout but not sure how to assert whether it is called three time on timeout.

Unit Test I tried:

[Fact]
public async Task Check_Whether_Given_Policy_Executed_OnTimeout()
{
    // Given / Arrange 
    IServiceCollection services = new ServiceCollection();

    bool retryCalled = false;

    HttpStatusCode codeHandledByPolicy = HttpStatusCode.InternalServerError;

   var data =  services.AddHttpClient<GraphQueryService>(Constants.PPQClient)
            .ConfigureHttpClient(client =>
            {
                client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                client.Timeout = TimeSpan.FromMilliseconds(Constants.ClientTimeOut);
            }).AddRetryPolicy()
    .AddHttpMessageHandler(() => new StubDelegatingHandler(codeHandledByPolicy));

 //Need to Check the retry logic is called three times. Not sure How to continue after this.

    Assert.Equal(codeHandledByPolicy, HttpStatusCode.InternalServerError);
    Assert.True(retryCalled);
}

Answer 1

Unfortunately you can't create unit tests to make sure that your policies have been setup properly. For example after you setup the retry count and sleep duration(s) you can't query them.

After reading the source code of Polly I've found a solution but it's super fragile since it relies on private fields. I've already raised a ticket, and it will be addressed in V8. (There is a huge uncertainty when it will be released.)

---

So, what can you do? Well, you can write integration tests where you are mocking the downstream http service. I've chosen the WireMock.Net library for this purpose.

I've created two abstractions over the downstream system:

FlawlessService

internal abstract class FlawlessServiceMockBase
{
    protected readonly WireMockServer server;
    private readonly string route;

    protected FlawlessServiceMockBase(WireMockServer server, string route)
    {
        this.server = server;
        this.route = route;
    }

    public virtual void SetupMockForSuccessResponse(IResponseBuilder expectedResponse = null, 
        HttpStatusCode expectedStatusCode = HttpStatusCode.OK)
    {
        server.Reset();

        var endpointSetup = Request.Create().WithPath(route).UsingGet();
        var responseSetup = expectedResponse ?? Response.Create().WithStatusCode(expectedStatusCode);

        server.Given(endpointSetup).RespondWith(responseSetup);
    }
}

and FautlyService

internal abstract class FaultyServiceMockBase
{
    protected readonly WireMockServer server;
    protected readonly IRequestBuilder endpointSetup;
    protected readonly string scenario;

    protected FaultyServiceMockBase(WireMockServer server, string route)
    {
        this.server = server;
        this.endpointSetup = Request.Create().WithPath(route).UsingGet();
        this.scenario = $"polly-setup-test_{this.GetType().Name}";
    }

    public virtual void SetupMockForFailedResponse(IResponseBuilder expectedResponse = null,
        HttpStatusCode expectedStatusCode = HttpStatusCode.InternalServerError)
    {
        server.Reset();

        var responseSetup = expectedResponse ?? Response.Create().WithStatusCode(expectedStatusCode);

        server.Given(endpointSetup).RespondWith(responseSetup);
    }

    public virtual void SetupMockForSlowResponse(ResilienceSettings settings, string expectedResponse = null)
    {
        server.Reset();

        int higherDelayThanTimeout = settings.HttpRequestTimeoutInMilliseconds + 500;

        server
            .Given(endpointSetup)
            .InScenario(scenario)
            //NOTE: There is no WhenStateIs
            .WillSetStateTo(1)
            .WithTitle(Common.Constants.Stages.Begin)
            .RespondWith(DelayResponse(higherDelayThanTimeout, expectedResponse));

        for (var i = 1; i < settings.HttpRequestRetryCount; i++)
        {
            server
                .Given(endpointSetup)
                .InScenario(scenario)
                .WhenStateIs(i)
                .WillSetStateTo(i + 1)
                .WithTitle($"{Common.Constants.Stages.RetryAttempt} #{i}")
                .RespondWith(DelayResponse(higherDelayThanTimeout, expectedResponse));
        }

        server
            .Given(endpointSetup)
            .InScenario(scenario)
            .WhenStateIs(settings.HttpRequestRetryCount)
            //NOTE: There is no WillSetStateTo
            .WithTitle(Common.Constants.Stages.End)
            .RespondWith(DelayResponse(1, expectedResponse));
    }

    private static IResponseBuilder DelayResponse(int delay) => Response.Create()
        .WithDelay(delay)
        .WithStatusCode(200);

    private static IResponseBuilder DelayResponse(int delay, string response) => 
        response == null 
            ? DelayResponse(delay) 
            : DelayResponse(delay).WithBody(response);
}

With these two classes you are able to simulate good and bad behaving downstream systems.

• The WireMock server will run locally on a specified port (details comes in a minute) and listens on a configurable route for GET requests
• ResilienceSettings is just a simple helper class to store timeout and retry policies' config values
• In case of faulty server, we have defined a scenario, which is basically a sequence of request-response pairs
  • In order to test the retry policy you can specify the number of intermediate steps
  • After all unsuccessful (intermediate) requests the WireMock server will transfer itself into an End state (WithTitle(Common.Constants.Stages.End)) and that's what you can query in your integration test

---

Here is a simple test which will issue request (with retries) against a slow downstream system. It fails several times but at the end it succeeds

[Fact]
public async Task GivenAValidInout_AndAServiceWithSlowProcessing_WhenICallXYZ_ThenItCallsTheServiceSeveralTimes_AndFinallySucceed()
{
    //Arrange - Proxy request
    HttpClient proxyApiClient = proxyApiInitializer.CreateClient();

    //Arrange - Service
    var xyzSvc = new FaultyXYZServiceMock(xyzServer.Value);
    xyzSvc.SetupMockForSlowResponse(resilienceSettings);

    //Act
    var actualResult = await CallXYZAsync(proxyApiClient);

    //Assert - Response
    const HttpStatusCode expectedStatusCode = HttpStatusCode.OK;
    actualResult.StatusCode.ShouldBe(expectedStatusCode);

    //Assert - Resilience Policy
    var logsEntries = xyzServer.Value.FindLogEntries(
        Request.Create().WithPath(Common.Constants.Routes.XYZService).UsingGet());
    logsEntries.Last().MappingTitle.ShouldBe(Common.Constants.Stages.End);
}

Please note that the proxyApiInitializer is an instance of a WebApplicationFactory<Startup> derived class.

And finally, this is how you can initialize your WireMock server

private static Lazy<WireMockServer> xyzServer;

public ctor()
{
   xyzServer = xyzServer ?? InitMockServer(API.Constants.EndpointConstants.XYZServiceApi);
}

private Lazy<WireMockServer> InitMockServer(string lookupKey)
{
    string baseUrl = proxyApiInitializer.Configuration.GetValue<string>(lookupKey);
    return new Lazy<WireMockServer>(
        WireMockServer.Start(new FluentMockServerSettings { Urls = new[] { baseUrl } }));
}