I have a section of code that runs if the user needs to re-auth after logging in. During UI tests, this popover is sometimes displayed, so I have a check for it existing
if (XCUIApplication().staticText["authLabel"].exists) {
completeAuthDialog()
}
When this runs locally, it is fine, completes and the framework finds the element no problem. But when the nightly job on the CI is ran, it fails the first time, but once the same build is set to be rebuilt, the test passes. authLabel is the UILabel's accessibility identifier(btw), so I have been trying to figure out what is causing the flickering.
Yesterday I spent time on the issue, and it seems that the framework just doesn't find the elements sometimes? I have used the accessibility inspector in ensure I am query for the same time it sees.
I even expanded that if check with 4 or 5 additional || to check for any element inside of the popover. The Elements all have accessibility identifiers, I have also used the record feature to ensure that it passes back the same element "names" I am using.
I am kind of stuck, I don't know what else to try/could be causing this issue. The worst part is it ran fine for couple of months, but it seems to fail every night now, and as I said when the tests are ran locally inside xcode they pass fine. Could this be a issue with building from command line?
It is often slower when your tests execute on a different machine, this problem seems particularly prevelant with CI machines as they tend to be under-powered.
If you just do a single check for an element existing, the test only has one point in time to get it right and if the app was slow to present the element then the test will fail.
You can defend against having a flaky test by using a waiter to check a few times over a few seconds to ensure that you've given the app enough time to show the authentication dialog before continuing.
let authElement = XCUIApplication().staticText["authLabel"]
let existsPredicate = NSPredicate(format: "exists == true")
let expectation = XCTNSPredicateExpectation(predicate: existsPredicate, object: authElement)
let result = XCTWaiter().wait(for: [expectation], timeout: 5)
if (result == .completed) {
completeAuthDialog()
}
You can adjust the timeout to suit your needs - a longer timeout will result in the test waiting a longer time to continue if the auth dialog doesn't appear, but will give the dialog more time to appear if the machine is slow. Try it out and see how flaky the tests are with different timeouts to optimise.