Detect errors and build report in case of failure in shake build

Question

I'm using shake for a test suite. I have multiples independent tests represented as a set of Rule. If any of theses rules fails, the test fails. Finally, I produce a report containing all the tests status.

My issues are :

a) I need to detect which test runs or fails. Actually I'm cheating using actionOnException, but that's lot of boilerplate in each command in each rules and that's complicated (I must write status file or play with IORef to store the failing status).

b) I want to write the Shake report as part of my final report, however shakeReport does not write the file in case of error, my only solution is to run again the build using --no-build --report out.html which is not convenient.

Edit: Actually the tests are in action and build their dependencies. The build roughly looks like :

main = do
  -- when this fails, `dumpTests` is called,
  -- but the shake report is not written
  _ <- (try shakeMain) :: IO (Either SomeException ())

  -- This write my test report from the success informations it can gather
  -- in the directory hierarchy
  dumpTests

smakeMain = shakeArgs {shakeStaunch=True, shakeReport=["report.html"]} $ do

   "tests" ~> need ["test1/done", "test2/done", ...]

   -- this rules completly runs a test
   "*/done" %> \done -> do
       let test = takeDirectory done
       -- clear all the leftover to be sure that there is nothing useless left. This is important because I use theses outputs to know which command succeeds or fails.
       liftIO $ removeFiles test ["stdout.log", "*/image/*.exr", "*/image/*.png", "done"]

       need [test </> "stdout.log"]

       results <- getDirectoryFiles (test </> "image") ["*.exr"]

       need (map (-<.> "png") results)

       writeFile' done "done"

   "*/stdout.log" %> \log -> do
        let path = takeDirectory log </> "test"
        need [path]

        aCmd path -- this builds stdout.log and all exrs

   "*/image/*.png" %> \png -> do
         need [(png -<.> "exr")]
         toExr png

Thank you.

Answer 1

Question b is simplest to answer. If the build raises an error it won't write out a report - it's reasonable that could be changed (I can see arguments both ways). However, you can automate the --no-build piece by having main call shake twice - first as you do now, and second with withArgs or using shake shakeOptions{shakeReport=...} mempty. Essentially you do exactly what you are doing now "by hand", but put it into the main function so it's automatic.

For the main question, I suspect the answer is you should be marking fails not with exceptions, but with a result value. For example, test1/done could record a TRUE/FALSE saying whether the test worked or not. Then you can have alldone which depends on all the */done values and writes a single report. That way your build would always pass (unless you got something fundamentally wrong), but the result on passing would be either "TESTS PASS" or "TESTS FAIL".