How to generate an [error] instead of an [info] upon seeing a wrong value in expect()

Question

Consider the following code:

import chisel3._
import chisel3.util._
import chisel3.iotesters._

class Inverter extends Module {
  val io = IO(new Bundle {
    val a = Input(UInt(4.W))
    val s = Output(UInt(4.W))
  })
  io.s := ~io.a
}

class InverterTester(c: Inverter) extends PeekPokeTester(c) {
  poke(c.io.a, 8)
  step(1)
  expect(c.io.s, 8) // Should be 7 here
}

object TestMain extends App {
  chisel3.iotesters.Driver.execute(args, () => new Inverter()) {
    c => new InverterTester(c)
  }
}

Now I run sbt 'test:runMain TestMain' and got this line (info is in purple):

[info] [0.002] EXPECT AT 1   io_s got 7 expected 8 FAIL

And the exit value of sbt is zero.

I need that line to be an error (with red color):

[error] [0.002] EXPECT AT 1   io_s got 7 expected 8 FAIL

As well as making the above sbt command exit with a non-zero value.

How can I achieve it with minimal change to existing code?

Answer 1

First the easy part. You can get a non-zero result code by using the result of chisel.execute like this.

  val result = chisel3.iotesters.Driver.execute(args, () => new Inverter()) {
    c => new InverterTester(c)
  }
  System.exit(if(result) 0 else 1)

Changing the logging level, unfortunately requires changing each of the separate backends in the chisel-testers repo. The following is an example of changing the TreadleBackend.scala, one of the three.

  def expect(signal: InstanceId, expected: BigInt, msg: => String)
    (implicit logger: TestErrorLog, verbose: Boolean, base: Int) : Boolean = {
    signal match {
      case port: Element =>
        val name = portNames(port)
        val got = treadleTester.peek(name)
        val good = got == expected

        if (!good) {
          logger error
                  s"""EXPECT AT $stepNumber $msg  $name got ${bigIntToStr(got, base)} expected ${bigIntToStr(expected, base)}""" +
                          s""" ${if (good) "PASS" else "FAIL"}"""
        }
        else if (verbose) {
          logger info
                  s"""EXPECT AT $stepNumber $msg  $name got ${bigIntToStr(got, base)} expected ${bigIntToStr(expected, base)}""" +
                          s""" ${if (good) "PASS" else "FAIL"}"""
        }
        if(good) treadleTester.expectationsMet += 1
        good
      case _ => false
    }
  }

This would not be an unreasonable issue to file, I think a logger.error would make more sense on an expect failure. There is some concern that changing this could have unexpected consequences for existing users, who are looking for that string.

But I'd like to encourage you to take a look at freechipsproject/chisel-testers2 repo. It's where the team is putting most of their testing development time. It would be easier to change, it has a lot of other nice features for building unit tests, and we are looking at ways we can make it better than chisel-testers.