How can I get repeatable async tests with FsCheck? Here is a sample code that I run in FSI:
let prop_simple() = gen {
let! s = Arb.generate<string>
printfn "simple: s = %A" s
return 0 < 1
}
let prop_async() =
async {
let s = Arb.generate<string> |> Gen.sample 10 1 |> List.head
// let! x = save_to_db s // for example
printfn "async: s = %A" s
return 0 < 1
}
|> Async.RunSynchronously
let check_props() =
//FC2.FsCheckModifiers.Register()
let config =
{ FsCheck.Config.Default with
MaxTest = 5
Replay = Random.StdGen(952012316,296546221) |> Some
}
Check.One(config, prop_simple)
Check.One(config, prop_async)
The output looks something like this:
simple: s = "VDm2JQs5z"
simple: s = "NVgDf2mQs8zaWELndK"
simple: s = "TWz3Yjl2tHFERyrMTvl0HOqgx"
simple: s = "KRWC92vBdZAHj6qcf"
simple: s = "CTJbQGXzpLBNn0RY6MCvlfUtbQhCUKm9tbXFhLSu0RcYmi"
Ok, passed 5 tests.
async: s = "aOE"
async: s = "y8"
async: s = "y8"
async: s = "q"
async: s = "q"
Ok, passed 5 tests.
Another run can look like this:
simple: s = "VDm2JQs5z"
simple: s = "NVgDf2mQs8zaWELndK"
simple: s = "TWz3Yjl2tHFERyrMTvl0HOqgx"
simple: s = "KRWC92vBdZAHj6qcf"
simple: s = "CTJbQGXzpLBNn0RY6MCvlfUtbQhCUKm9tbXFhLSu0RcYmi"
Ok, passed 5 tests.
async: s = "g"
async: s = "g"
async: s = "g"
async: s = ""
async: s = ""
Ok, passed 5 tests.
So prop_simple()
works fine & is repeatable (given StdGen(952012316,296546221)
).
But prop_async()
is not repeatable & seems to generate the same strings over and over.
Also, is there a better way to write prop_async()
?
FsCheck's behavior doesn't really have anything to do with the async
here, but rather with the fact that inside the async
you're using Gen.sample
. Gen.sample
picks a new time-based seed for every invocation - so the behavior of it insides an FsCheck property is not reproducible. In other words you shouldn't ever use it inside a property, it's there just for exploratory purposes when you're writing a new generator. Since the seed is time-based, and your property is very small, multiple invocations will use the same seed and so you see the same values. As an example, here is a property without any async
that has the same behavior:
let prop_simple2() =
let s = Arb.generate<string> |> Gen.sample 10 1 |> List.head
// let! x = save_to_db s // for example
printfn "simple2: s = %A" s
0 < 1
prints e.g.
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
Ok, passed 5 tests.
Now as for how to write an async
property, I'd keep the asynchrony inside the property and then resolve it using Async.RunSynchronously
to a normal value.As a variant on your example:
let prop_async2 =
gen {
let! s = Arb.generate<string>
// let! x = save_to_db s // for example
let r =
async {
printfn "async2: s = %A" s
}
|> Async.RunSynchronously
return 0 < 1
}
Which has a deterministic output. (Note also if you're already creating a Gen<'T>
instance you don't need to make the property a function. You can, but that just means that FsCheck will generate 100 values for the unit
type (these values are of course all ()
which is effectively null
, so it doesn't hurt but is a small performance improvement.)
You can also do it the other way round:
let prop_async3 =
async {
let r = gen {
let! s = Arb.generate<string>
printfn "async3: s = %A" s
return 0 < 1
}
return r
}
|> Async.RunSynchronously
A few gotchas to be aware of.
Sequential asynchronous code should generally pose little problems, but read on.
Asynchronous and concurrent code may run into issues like munn said in the comments, i.e. multiple threads/tasks using the same value. Also reproducibility will be affected. You can maybe carefully write your property code so you don't run into this (e.g. by having a prelude in your properties where all necessary values are first generated in a sequential fashion, then kick off the asynchronous functions), but it needs some work and thought.
If you override Arbitrary
instances using Arb.register
they will be overridden in a thread local way; i.e. they won't be propagated to a sequence of async Task
s. My advice is to just don't do that. Registered Arbitrary
instances are essentially mutable static state and that generally doesn't play all that nice with concurrency.
Taken together I think async
properties are definitely possible but it's definitely somehwat of an uphill battle in v2. FsCheck 3 (currently in alpha) supports async and multithreaded execution directly.