`testthat::expect_silent()` does not seem to notice ggplot2 errors

Question

I'm having trouble understanding the following behaviour of the expect_silent() function from testthat.

expect_silent() is supposed to fail when the test code returns any output, for example an error or warning:

library(testthat)

test_that("expect_silent works as expected", {
  expect_silent( {
    stop()
  } )
} )
#> Error: Test failed: 'expect_silent works as expected'
#> * 
#> 1: expect_silent({
#>        stop()
#>    }) at <text>:5
#> 2: quasi_capture(enquo(object), evaluate_promise)
#> 3: capture(act$val <- eval_bare(get_expr(quo), get_env(quo)))
#> 4: withr::with_output_sink(temp, withCallingHandlers(withVisible(code), warning = handle_warning, 
#>        message = handle_message))
#> 5: force(code)
#> 6: withCallingHandlers(withVisible(code), warning = handle_warning, message = handle_message)
#> 7: withVisible(code)
#> 8: eval_bare(get_expr(quo), get_env(quo))
#> 9: stop() at <text>:6

(The above is the expected behaviour: expect_silent() detects the error produced by stop(), and the test fails.)

However, for some reason it doesn't seem to detect errors that occur in ggplot2 expressions. For example, the following ggplot2 code produces an error due to a misspelling:

library(ggplot2)

ggplot(diamonds, aes(x = carrot, y = price)) +
  geom_point()
#> Error in FUN(X[[i]], ...): object 'carrot' not found

But expect_silent() doesn't seem to detect the error:

test_that("expect_silent fails when ggplot2 throws an error", {
  expect_silent( {
    ggplot(diamonds, aes(x = carrot, y = price)) +
      geom_point()
  } )
} )

(No output is produced.)

Am I misunderstanding the purpose of expect_silent()? This is causing me a real headache as I'm trying to use it to test a ggplot2 extension.

Answer 1

Try capturing the output from ggplot and then testing if it can be printed:

library(ggplot2)
library(testthat)

# First test should succeed (no output)
test_that("silent when ggplot2 succeeds", {
  working.plot <- ggplot(diamonds, aes(x = carat, y = price)) + geom_point()
  expect_silent(print(working.plot))
} )

# Second test should fail
test_that("fails when ggplot2 throws an error", {
  broken.plot <- ggplot(diamonds, aes(x = carrot, y = price)) + geom_point()
  expect_silent(print(broken.plot))
} )

The second test fails with copious output which I've curtailed below:

Error: Test failed: 'expect_silent fails when ggplot2 throws an error'
* object 'carrot' not found

Update - 15th Dec 2018

Regarding your comment about why print() is necessary:

The ggplot() function returns an object of class ggplot. The ggplot2 package overloads the print() function, so instead of printing the object to STDOUT in the R session terminal, it prints the chart. The interactive mode in the R session terminal assumes that most of the commands are run through the print() function.

The testthat tests are evaluated in their own environments. The testthat environments are non-interactive, so the running through the print() function assumption no longer holds. You can test this with the interactive() function that comes with base R. It should report TRUE in the R session terminal and FALSE within a test_that() call.