Customise FsCheck output

I am testing with FsCheck and NUnit in VisualStudio.

The problem currently is: I managed to generate random graphs (for testing some graph functionality) but when a test fails, FsCheck spits out the whole graph and it does not use ToString so it literally dumps raw list of records and you cannot see anything in there.

Also I would need not only the input graph for inspection but also some other data that I create when running a property.

So how can I change the output behaviour of FsCheck in order to

actually call my ToString method on the input graph
output further information

when a test fails?

EDIT: Here is my current test setup.

module GraphProperties

open NUnit.Framework
open FsCheck
open FsCheck.NUnit

let generateRandomGraph =
    gen {
        let graph: Graph<int,int> = Graph<_,_>.Empty()
        // fill in random nodes and transitions...
        return graph
    }

type MyGenerators =
    static member Graph() =
        {new Arbitrary<Graph<int,int>>() with
            override this.Generator = generateRandomGraph
            override this.Shrinker _ = Seq.empty }

[<TestFixture>]
type NUnitTest() =
    [<Property(Arbitrary=[|typeof<MyGenerators>|], QuietOnSuccess = true)>]
    member __.cloningDoesNotChangeTheGraph (originalGraph: Graph<int,int>) =
        let newGraph = clone originalGraph
        newGraph = originalGraph

Solution

FsCheck uses sprintf "%A" to convert test parameters to strings in the test output, so what you need to do is control how your types are formatted by the %A formatter. According to How do I customize output of a custom type using printf?, the way to do that is with the StructuredFormatDisplay attribute. The value of that attribute should be a string in the format PreText {PropertyName} PostText, where PropertyName should be a property (not a function!) on your type. E.g., let's say you have a tree structure with some complicated information in the leaves, but for your testing you only need to know about the number of leaves, not what's in them. So you'd start with a data type like this:

// Example 1
type ComplicatedRecord = { ... }
type Tree =
    | Leaf of ComplicatedRecord
    | Node of Tree list
    with
        member x.LeafCount =
            match x with
            | Leaf _ -> 1
            | Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
        override x.ToString() =
            // For test output, we don't care about leaf data, just count
            match x with
            | Leaf -> "Tree with a total of 1 leaf"
            | Node -> sprintf "Tree with a total of %d leaves" x.LeafCount

Now, so far that's not what you want. This type does not have a custom %A format declared, so FsCheck (and anything else that uses sprintf "%A" to format it) will end up outputting the whole complicated structure of the tree and all its irrelevant-to-the-test leaf data. To make FsCheck output what you want to see, you'll need to set up a property, not a function (ToString won't work for this purpose) that will output what you want to see. E.g.:

// Example 2
type ComplicatedRecord = { ... }
[<StructuredFormatDisplay("{LeafCountAsString}")>]
type Tree =
    | Leaf of ComplicatedRecord
    | Node of Tree list
    with
        member x.LeafCount =
            match x with
            | Leaf _ -> 1
            | Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
        member x.LeafCountAsString = x.ToString()
        override x.ToString() =
            // For test output, we don't care about leaf data, just count
            match x with
            | Leaf -> "Tree with a total of 1 leaf"
            | Node -> sprintf "Tree with a total of %d leaves" x.LeafCount

NOTE: I haven't tested this in F#, just typed it into the Stack Overflow comment box -- so it's possible that I've messed up the ToString() part. (I don't remember, and can't find with a quick Google, whether overrides should be after or before the with keyword). But I know that the StructuredFormatDisplay attribute is what you want, because I've used this myself to get custom output out of FsCheck.

By the way, you could also have set a StructuredFormatDisplay attribute on the complicated record type in my example as well. For example, if you have a test where you care about the tree structure but not about the contents of the leaves, you'd write it like:

// Example 3
[<StructuredFormatDisplay("LeafRecord")>] // Note no {} and no property
type ComplicatedRecord = { ... }
type Tree =
    | Leaf of ComplicatedRecord
    | Node of Tree list
    with
        member x.LeafCount =
            match x with
            | Leaf _ -> 1
            | Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
        override x.ToString() =
            // For test output, we don't care about leaf data, just count
            match x with
            | Leaf -> "Tree with a total of 1 leaf"
            | Node -> sprintf "Tree with a total of %d leaves" x.LeafCount

Now all your ComplicatedRecord instances, no matter their contents, will show up as the text LeafRecord in your output, and you'll be better able to focus on the tree structure instead -- and there was no need to set a StructuredFormatDisplay attribute on the Tree type.

This isn't a totally ideal solution, as you might need to adjust the StructuredFormatDisplay attribute from time to time, as needed by the various tests you're running. (For some tests you might want to focus on one part of the leaf data, for others you'd want to ignore the leaf data entirely, and so on). And you'll probably want to take the attribute out before you go to production. But until FsCheck acquires a "Give me a function to format failed test data with" config parameter, this is the best way to get your test data formatted the way you need it.