Options to execute unit Test cases in minimum Time for Time Based Unit Test cases

Question

I have a requirement to lockout a user if three failed attempts are made within 15 minutes. The account will be automatically unlocked after a period. Now I am passing the parameters - maximum attempt count, the lockout window duration and lockout period as parameters to the class that implements the functionality. Even with values like 2s or 3s for the parameters will result in the unit test suite execution to complete about 30 seconds.

Is there any specific method or strategies used in these scenarios to reduce the test execution time?

Answer 1

There are a couple of options:

• Use a Test Double and inject an IClock that a test can control.
• Use a smaller time resolution than seconds. Perhaps define the window and the quarantine period in milliseconds instead of seconds.
• Write the logic as one or more pure functions.

Pure functions are intrinsically testable, so let me expand on that.

Pure functions

In order to ensure that we're working with pure functions, I'll write the tests and the System Under Test in Haskell.

I'm going to assume that some function exists that checks whether a single login attempt succeeds. How this works is a separate concern. I'm going to model the output of such a function like this:

data LoginResult = Success | Failure deriving (Eq, Show)

In other words, a login attempt either succeeds or fails.

You now need a function to determine the new state given a login attempt and a previous state.

Evaluate login

At first I thought this was going to be more complicated, but the nice thing about TDD is that once you write the first tests, you realise the potential for simplification. Fairly quickly, I realised that all that was required was a function like this:

evaluateLogin :: (UTCTime, LoginResult) -> [UTCTime] -> [UTCTime]

This function takes a current LoginResult and the time it was made (as a tuple: (UTCTime, LoginResult)), as well as a log of previous failures, and returns a new failure log.

After a few iterations, I'd written this inlined HUnit parametrised test:

"evaluate login" ~: do
  (res, state, expected) <-
    [
      ((at 2022 5 16 17 29, Success), [],
        [])
      ,
      ((at 2022 5 16 17 29, Failure), [],
        [at 2022 5 16 17 29])
      ,
      ((at 2022 5 16 18 6, Failure), [at 2022 5 16 17 29],
        [at 2022 5 16 18 6, at 2022 5 16 17 29])
      ,
      ((at 2022 5 16 18 10, Success), [at 2022 5 16 17 29],
        [])
    ]
  let actual = evaluateLogin res state
  return $ expected ~=? actual

The logic I found useful to tease out is that whenever there's a test failure, the evaluateLogin function adds the failure time to the failure log. If, on the other hand, there's a successful login, it clears the failure log:

evaluateLogin :: (UTCTime, LoginResult) -> [UTCTime] -> [UTCTime]
evaluateLogin (   _, Success)          _ = []
evaluateLogin (when, Failure) failureLog = when : failureLog

This, however, tells you nothing about the quarantine status of the user. Another function can take care of that.

Quarantine status

The following parametrised tests is the result of a few more iterations:

"is locked out" ~: do
  (wndw, p, whn, l, expected) <-
    [
      (ndt 0 15, ndt 1 0, at 2022 5 16 19 59, [], False)
      ,
      (ndt 0 15, ndt 1 0, at 2022 5 16 19 59, [
        at 2022 5 16 19 54,
        at 2022 5 16 19 49,
        at 2022 5 16 19 45
      ],
        True)
      ,
      (ndt 0 15, ndt 1 0, at 2022 5 16 19 59, [
        at 2022 5 16 19 54,
        at 2022 5 16 19 49,
        at 2022 5 16 18 59
      ],
        False)
      ,
      (ndt 0 15, ndt 1 0, at 2022 5 16 19 59, [
        at 2022 5 16 19 54,
        at 2022 5 16 19 52,
        at 2022 5 16 19 49,
        at 2022 5 16 19 45
      ],
        True)
      ,
      (ndt 0 15, ndt 1 0, at 2022 5 16 20 58, [
        at 2022 5 16 19 54,
        at 2022 5 16 19 49,
        at 2022 5 16 19 45
      ],
        False)
    ]
  let actual = isLockedOut wndw p whn l
  return $ expected ~=? actual

These tests drive the following implementation:

isLockedOut :: NominalDiffTime -> NominalDiffTime -> UTCTime -> [UTCTime] -> Bool
isLockedOut window quarantine when failureLog =
  case failureLog of
    [] -> False
    xs ->
      let latestFailure = maximum xs
          windowMinimum = addUTCTime (-window) latestFailure
          lockOut = 3 <= length (filter (windowMinimum <=) xs)
          quarantineEndsAt = addUTCTime quarantine latestFailure
          isStillQuarantined = when < quarantineEndsAt
      in
        lockOut && isStillQuarantined

Since it's a pure function, it can calculate quarantine status deterministically based exclusively on input.

Determinism

None of the above functions depend on the system clock. Instead, you pass the current time (when) as an input value, and the functions calculate the result based on the input.

Not only is this easy to unit test, it also enables you to perform simulations (a test is essentially a simulation) and calculate past results.