ScalaCheck, different behaviour when calling Test.check or Test.checkProperties

Question

I have the following code that uses ScalaCheck properties to test some class

package quickcheck.tests

import org.scalacheck.Arbitrary._
import org.scalacheck.Gen._
import org.scalacheck.Prop._
import org.scalacheck._
import org.scalacheck.util.ConsoleReporter

class Heap
{
}

object Heap
{
  val empty = new Heap
  def insert(i: Int, h: Heap) = new Heap
}

class TestCheck extends Properties("Heap")
{
  lazy val genHeap: Gen[Heap] =
  {
    def sizedHeap(size: Int): Gen[Heap] =
    {
      if (size <= 0)
        Heap.empty
      else
        for
        (
          i <- arbitrary[Int] suchThat(_ > Int.MinValue);
          s <- choose(0, size);
          h <- sizedHeap(s)
        )
          yield
            Heap.insert(i, h)
    }

    Gen.sized[Heap](sizedHeap)
  }

  implicit lazy val arbHeap: Arbitrary[Heap] = Arbitrary(genHeap)

  property("test1") = forAll
  {
    (h: Heap) => true
  }

  property("test2") = forAll
  {
    (h1: Heap, h2: Heap, n: Int) => true
  }
}

object MyTest extends App
{
  println("*** TEST 1")
  val checkHeap = new TestCheck
  Test.checkProperties(Test.Parameters.default.withTestCallback(ConsoleReporter(1)), 
    checkHeap)

  println("*** TEST 2")
  val checkHeap2 = new TestCheck
  checkHeap2.check

  println("*** TEST 3")
  val checkHeap3 = new TestCheck
  Test.check(Test.Parameters.default.withTestCallback(ConsoleReporter(1)),     checkHeap)
}

If I run it thru the ScalaCheck Test class I get different results if I use method Test.checkProperties or method Test.check.

This is the output I get:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
! Gave up after only 40 passed tests. 202 tests were discarded.

My questions is why TEST1 gives different result than TEST3.

If I remove the suchThat filter and leave the for statement in the sizeHead method as this:

for
(
  i <- arbitrary[Int]
  s <- choose(0, size);
  h <- sizedHeap(s)
)
  yield
    Heap.insert(i, h)

I get the following result:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
+ OK, passed 100 tests.

Is this a bug or it is the correct behaviour? Keep in mind that ScalaTest Checkers.check uses Test.check.

Answer 1

You get different results because you're doing different things. Your first two tests—which are effectively the same—check all properties separately, whereas your third one tests all properties as if they were a single property.

Take a look at the signatures of Test.check and Test.checkProperties: The former takes a single Prop, whereas the latter takes a Properties container.

In scalacheck 1.12 Properties inherits from Prop; if you pass Properties as Prop you get a new property that tests whether all contained properties hold. The consequence is that you test all your TestCheck properties with a single check configuration (i.e. generators, iteration limits, etc.), and naturally that check configuration is exhausted at some point.

So yes, that's expected behaviour. Of course, it's absolutely confusing, hence scalacheck 1.13 removed this feature: Properties does not inherit from Prop anymore; your example would not compile on scalacheck 1.13 anymore.