Cypress 12 spec / test execution order is not stable / flaky tests

Question

We upgraded from Cypress 10 to Cypress 12. On Cypress 10 all tests were green, and not very flaky (we executed the GitLab pipeline before the upgrade ~ 5 times without any code changes and all was green).

Now after the upgrade, some tests fail inconsistently.

Also we notices they are executed in a slightly different - but not fully random order.

We know that tests should not fail depending on execution order normally. But it would be really helpful to have a way to execute them in a consistent order to at least be able to debug.

Also - if this would be a new feature of Cypress 12 to execute tests randomly, I would expect more randomness (see screenshots)

Run 1:

Run 2:

EDIT: It prints:

Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}

and the found specs. Already here the order is different. My first guess is that the glob function used here is no longer stable?

EDIT2: It did a quick check with the glob package locally - the order seems to be stable here.

Answer 1

Please see Cypress 10 - How to run all tests in one go, you should be able to control the execution order exactly by creating a controlling spec that imports the specs you want to run.

// ordered-specs.cy.js

import './test1.spec.cy.js'    // relative paths
import './test2.spec.cy.js'
...

npx cypress run --spec cypress/e2e/ordered-specs.cy.js

Another option would be the Module API to execute specs individually and in sequence. Since this is a node script you can use fs tools to read the test folders and pick up current tests without editing the script.

There also this answer Filter tests using "Run all tests" in Cypress version 10+ where you can generate a controlling test. It still needs to be regenerated when new tests are added but you can apply the same logic at the top of your Module API script if you wish.