How to capture last trial result in Poly retry policy?

Question

I am working on extending the TestMethod attribute in .NET Core. I am using Polly library for retry logic along with an outer timeout policy.

I want a helper method that can retry invoking the ITestMethod until it passes. I don't mind the number of times it'll be retried. But I'll be putting a timeout within which it has to be completed. If the delegate is executed successfully within timeout, it's fine. But if there is a timeout exception, I still want the failed result value (the result of the last iteration) instead of TimeOutRejectedException or the default value of the return type.

Below is my extended test method attribute class:

public sealed class GuaranteedPassTestMethodAttribute : TestMethodAttribute
{
    /// <inheritdoc/>
    public override TestResult[] Execute(ITestMethod testMethod)
    {
        return ExecuteTestTillSuccess(testMethod);
    }

    private TestResult[] ExecuteTestTillSuccess(ITestMethod testMethod)
    {
        var gracefulTestRun =
            TestExecutionHelper.ExecuteTestTillPassAsync(
                () => TestInvokeDelegate(testMethod));

        return gracefulTestRun.Result;
    }

    private Task<TestResult[]> TestInvokeDelegate(ITestMethod testMethod)
    {
        TestResult[] result = null;
        var thread = new Thread(() => result = Invoke(testMethod));
        thread.Start();
        thread.Join();

        return Task.FromResult(result);
    }
}

Below is my TestExecutionHelper that uses Polly:

internal static class TestExecutionHelper
{
    private static readonly Func<TestResult[], bool> TestFailurePredicate =
        results => results != null &&
                   results.Length == 1 &&
                   results.First().Outcome != UnitTestOutcome.Passed;

    internal static async Task<TestResult[]> ExecuteTestTillPassAsync(
        Func<Task<TestResult[]>> testInvokeDelegate,
        int delayBetweenExecutionInMs = 3000,
        int timeoutInSeconds = 60 * 10)
    {
        var timeoutPolicy = Policy.TimeoutAsync<TestResult[]>(timeoutInSeconds);
        var retryPolicy = Policy.HandleResult<TestResult[]>(TestFailurePredicate)
                                .WaitAndRetryAsync(int.MaxValue, x => TimeSpan.FromMilliseconds(delayBetweenExecutionInMs));
        var testRunPolicy = timeoutPolicy.WrapAsync(retryPolicy);
        return await testRunPolicy.ExecuteAsync(testInvokeDelegate);
    }
}

With this setup, I either get a passed test method execution or TimeOutRejectedException for failed tests. I want to capture failed test's TestResult even after retries.

Answer 1

Test case

Let's suppose we have the following test:

[TestClass]
public class UnitTest1
{
    private static int counter;
    [GuaranteedPassTestMethod]
    public async Task TestMethod1()
    {
        await Task.Delay(1000);
        Assert.Fail($"Failed for {++counter}th time");
    }
}

I've used a static variable (called counter) to change the output of each test run.

Attribute

I've simplified your attribute's code by making use of the Task.Run:

public sealed class GuaranteedPassTestMethodAttribute : TestMethodAttribute
{
    public override TestResult[] Execute(ITestMethod testMethod)
        => TestExecutionHelper
            .ExecuteTestTillPassAsync(
                async () => await Task.Run(
                    () => testMethod.Invoke(null)))
            .GetAwaiter().GetResult();
}

I've also changed the .Result to GetAwaiter().GetResult() for better exception handling.

• If you are not familiar with this pattern please read this topic.

Strategy

I've introduced an accumulator variable called Results where I capture all the test runs' results.

internal static class TestExecutionHelper
{
    private static readonly List<TestResult> Results = new List<TestResult>();
    private static readonly Func<TestResult, bool> TestFailurePredicate = result =>
    { 
        Results.Add(result);
        return result != null && result.Outcome != UnitTestOutcome.Passed;
    };

    internal static async Task<TestResult[]> ExecuteTestTillPassAsync(
        Func<Task<TestResult>> testInvokeDelegate,
        int delayBetweenExecutionInMs = 3000,
        int timeoutInSeconds = 1 * 10)
    {
        var timeoutPolicy = Policy.TimeoutAsync<TestResult>(timeoutInSeconds);
        var retryPolicy = Policy.HandleResult(TestFailurePredicate)
            .WaitAndRetryForeverAsync(_ => TimeSpan.FromMilliseconds(delayBetweenExecutionInMs));
        var testRunPolicy = timeoutPolicy.WrapAsync(retryPolicy);

        try { await testRunPolicy.ExecuteAsync(testInvokeDelegate); }
        catch (TimeoutRejectedException) { } //Suppress

        return Results.ToArray();
    }
}

• I've used here WaitAndRetryForeverAsync instead of WaitAndRetryAsync(int.MaxValue, ....
• I've also change the signature of the testInvokeDelegate to align to the interface.

Output

For my test I've decreased the default value of timeoutInSeconds to 10 seconds.

 TestMethod1
   Source: UnitTest1.cs line 17

Test has multiple result outcomes
   3 Failed

Results

    1)  TestMethod1
      Duration: 1 sec

      Message: 
        Assert.Fail failed. Failed for 1th time
      Stack Trace: 
        UnitTest1.TestMethod1() line 20
        ThreadOperations.ExecuteWithAbortSafety(Action action)

    2)  TestMethod1
      Duration: 1 sec

      Message: 
        Assert.Fail failed. Failed for 2th time
      Stack Trace: 
        UnitTest1.TestMethod1() line 20
        ThreadOperations.ExecuteWithAbortSafety(Action action)

    3)  TestMethod1
      Duration: 1 sec

      Message: 
        Assert.Fail failed. Failed for 3th time
      Stack Trace: 
        UnitTest1.TestMethod1() line 20
        ThreadOperations.ExecuteWithAbortSafety(Action action)

Let's see the timeline for this test run:

• 0 >> 1: TestMethod1's Delay
• 1: Assert.Fail
• 1 >> 4: Retry's penalty
• 4 >> 5: TestMethod1's Delay
• 5: Assert.Fail
• 5 >> 8: Retry's penalty
• 8 >> 9: TestMethod1's Delay
• 9: Assert.Fail
• 9 >> 12: : Retry's penalty
• 10: Timeout kicks in

There is one thing to note here: The Test Duration won't contain the retry's penalty delays: