I'm playing with FsCheck so I have this implementation:
let add a b =
if a > 100
then failwith "nasty bug"
else a + b
...and this FsCheck based test:
fun (a:int) -> (add a 0) = a
|> Check.QuickThrowOnFailure
and the test never fails. My guess is that the 100 values produced by the random generator are never bigger than 100.
Shouldn't the values be more "random"?
When you use Check.QuickThrowOnFailure, it uses the configuration Config.QuickThrowOnFailure, which has these values:
> Config.QuickThrowOnFailure;;
val it : Config =
{MaxTest = 100;
MaxFail = 1000;
Replay = null;
Name = "";
StartSize = 1;
EndSize = 100;
QuietOnSuccess = false;
Every = <fun:get_Quick@342>;
EveryShrink = <fun:get_Quick@343-1>;
Arbitrary = [];
Runner = <StartupCode$FsCheck>.$Runner+get_throwingRunner@355;}
The important values to consider here are StartSize, but particularly EndSize. Some of the generators in FsCheck uses the size context to determine the size or range of values it generates.
If you change the EndSize to e.g. 1,000 you can make your test fail:
> Check.One({Config.QuickThrowOnFailure with EndSize = 1000}, fun (a:int) -> (add a 0) = a);;
System.Exception: Falsifiable, after 15 tests (0 shrinks) (StdGen (1912816373,296229213)):
Original:
101
with exception:
> System.Exception: nasty bug
at FSI_0040.add(Int32 a, Int32 b)
at [email protected](Int32 a)
at FsCheck.Testable.evaluate[a,b](FSharpFunc`2 body, a a) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Testable.fs:line 161
at <StartupCode$FsCheck>[email protected](String message) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
at <StartupCode$FsCheck>[email protected](String , TestResult ) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
at FsCheck.Runner.check[a](Config config, a p) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 275
at <StartupCode$FSI_0055>.$FSI_0055.main@()
Stopped due to error