Search code examples
haskellquickcheckproperty-based-testinghaskell-hedgehog

Example that shows the limitations of integrated shrinking


I just watched a video that presents the notion of integrated shrinking for property based tests. The approach seems to have some advantages over type directed shrinking, however it was pointed out in this reddit thread that the integrated shrinking approach does not fit well in the case of monadic generators:

Doing shrinking in your way does not fit well with a monadic style for generators. Here is an example, consider generating an arbitrary list (ignore termination for now):

do x <- arbitrary
   xs <- arbitrary
   return (x:xs)

Now, the default behavior of your shrinking would first shrink x (holding xs constant), and then shrink xs (holding x constant), which severely limits the shrinking (the concept of local minimum is now a lot less strong).

I read the above comment as "integrated shrinking might fail to provide a global minimum counter example". However, since hedgehog seems to be able to find minimal counter examples for failed properties on lists, I was wondering if there is an example that could show the drawback pointed out in the quote above.


Solution

  • In calculus terms, the problem is that you aren't following the negative gradient (steepest descent), instead you're minimizing along 1 axis first and then minimizing along the other axis. Based on this analogy, it is easy to come up with at least a contrived example - consider the function

    -- f x y = ((x^2 - 1)^2 - 0.2*x) * ((y^2 - 1/2)^2 - 0.1*y)
    f x y = (x^4 - 2.2*x^2 + 1) * (y^4 - 1.1*y^2 + 1/4)
    

    See plot on WolframAlpha.

    and we're testing it for the property f x y > 0, and let's say a minimal example would have a point closest to the origin (0, 0). Depending on where you first start shrinking, it is entirely possible that you end up close to (±1, 0) because you adjust x first and then don't allow y to change much. However, in an ideal situation, you'd want to end up somewhere close to (0, ±1/2) to satisfy the minimality criterion.