How to ignore benchmarks when not using nightly?

Question

I have a file with some benchmarks and tests and would like to test against stable, beta and nightly. However, either I don't use the benchmark or stable/beta complain. Is there a way to hide all the benchmark parts when using stable/beta?

As an example the following code from the book:

#![feature(test)]

extern crate test;

pub fn add_two(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    use super::*;
    use test::Bencher;

    #[test]
    fn it_works() {
        assert_eq!(4, add_two(2));
    }

    #[bench]
    fn bench_add_two(b: &mut Bencher) {
        b.iter(|| add_two(2));
    }
}

I'm using rustup and would like the same file to work with all the builds, calling something like:

rustup run nightly cargo bench --bin bench --features "bench"
rustup run nightly cargo test --bin bench --features "bench"
rustup run beta cargo test --bin bench
rustup run stable cargo test --bin bench

I was able to hide the #![feature(test)] with #![cfg_attr(feature = "bench", feature(test))]. Can I do something similar to the rest of the benchmark parts? What is a good resource for feature flags?

Answer 1

In my projects, I place benchmarks in a separate module, just like I do for tests. I then create a Cargo feature that enables them. In this excerpt, I used the feature name unstable, but you can use anything you'd like:

Cargo.toml

# ...

[features]
unstable = []

# ...

src/lib.rs

#![cfg_attr(feature = "unstable", feature(test))]

#[cfg(test)]
mod tests {
    #[test]
    fn a_test() {
        assert_eq!(1, 1);
    }
}

#[cfg(all(feature = "unstable", test))]
mod bench {
    extern crate test;
    use self::test::Bencher;

    #[bench]
    fn a_bench(b: &mut Bencher) {
        let z = b.iter(|| {
            test::black_box(|| {
                1 + 1
            })
        });
    }
}

The line #[cfg(all(feature = "unstable", test))] says to only compile the following item if the feature is set and we are compiling in test mode anyway. Likewise, #![cfg_attr(feature = "unstable", feature(test))] only enables the test feature flag when the unstable feature is enabled.

Here's an example in the wild.