I have a Criterion benchmark where each bgroup corresponds to a test, and within each bgroup there are two bench values of the test with different options. For example:
main = defaultMain
[bgroup "test1" [bench "v1" test1_1, bench "v2" test1_2]
,bgroup "test2" [bench "v1" test2_1, bench "v2" test2_2
-- lots more tests
]
Within each bgroup the two bench tests are comparable. However, test1 takes 2000 micro seconds, while test2 takes 45 micro seconds. The overview graph (which is most useful for what I want to do) displays different tests on the same axes, so I can clearly see the differences in test1, but test2 is hard to see.
Is it possible to normalise each bgroup for plotting? Or show them on separate axes? Or should I dump the CSV data and plot what I want myself?
This issue is definitely amongst the shortcomings of Criterion. I've been bitten by the same problem multiple times.
The standard approach I take to work around this is just to generate an individual executable per each comparison unit. A special benchmark target has been added in the latest versions of Cabal, so I declare a benchmark target per each comparison unit in the .cabal file. Then I can run each comparison using cabal bench [target-name]. Yeah, it is all far from comforting, but it's the best I could come up with.