Search code examples
javaswingunit-testinggui-testingassertj

Why do we get occasional failures in Assertj Swing tests?

Our gui unit-tests with Assertj Swing 3.9.2 occasionally fail in a difficult to reproduce way. Sometimes the entire test suite is green, sometimes some of the test cases fail. We are using two different machines with Ubuntu 18.04 LTS and GNOME and we get the same errors.

The exceptions thrown are two and appear from time to time in different test cases:

  • org.assertj.swing.exception.ComponentLookupException
  • org.assertj.swing.exception.WaitTimedOutError

What is giving us troubles is that this behavior seems totally random, and the test suite behaves differently depending on the following factors:

  • Execution from Eclipse, with Eclipse as full window.
  • Execution from Eclipse, with Eclipse occupying a small fraction of the screen (so as the windows created from the tests do not overlap with Eclipse itself or other windows).
  • Execution from Maven.
  • Execution of single test cases or entire suite.

Furthermore, re-running multiple times the same tests could bring to failures or success in different test cases.

The tests also failed on Travis-Ci.

As additional information we can include that for ComponentLookupException, the component not found is actually showed on the screen but the component hierarchy is empty, as can be seen in this excerpt from a Travis build:

...

970 org.assertj.swing.exception.ComponentLookupException: 
971 Unable to find component using matcher org.assertj.swing.core.NameMatcher[name='labelTextField', type=javax.swing.text.JTextComponent, requireShowing=true].
972
973 Component hierarchy:
974 apt.project.frontend.view.swing.CustomDialog[name='dialog1', title='', enabled=true, modal=false, visible=true, showing=true]
975
976 at apt.project.frontend.view.swing.CustomDialogTest.testWhenOkButtonIsClickedThenInputIsSavedBeforeClosing(CustomDialogTest.java:62)

...

As for the WaitTimedOutError exception, when the test fails, it is because the component is not actually shown on the screen (the window does not physically appear).

You can further inspect the error log and the code itself in wiew.swing package in our GitHub repository and in this Travis build

We have not been able to understand the causes of these issues as they seem totally random.

Is this a somehow known behavior or have we implemented the tests (CoursePanelTest, CustomDialogTest) in a wrong manner?

Solution

  • I see from the Travis-CI log that you are using Xvfb. I had similar issues using that. Switching to VNC, as recommended by AssertJ (https://joel-costigliola.github.io/assertj/assertj-swing-running.html) fixed them.