Search code examples
puppeteercypresse2e-testingplaywright

What's the benefit of E2E testing only against latest browser versions

When performing end-to-end tests with tools such as Playwright, Puppeteer and Cypress, I believe all of them (except the latter, pardon me if I'm wrong) only allow you to use the very latest version of each browser which is bundled along with every release.

In that case, I was wondering what would be the real benefit of running your acceptance end-to-end tests which would only validate functionality against a more limited scenario instead of, for example, making sure your application would still work on Chrome 70?

Any insights about why considering only recent browser versions and forgetting about the rest is a good idea, by using Playwright instead of a tool which would allow targeting tests against a specific browser binary, for example and have your E2E testing resulting like:

PASS. Shopping-Cart (Modern)
FAIL. Shopping-Cart (chrome 70) 
// Refactor

instead of

PASS. Shopping-Cart
// Yay! Release
Solution

  • Puppeteer, Playwright, Cypress with specific executable

    First, let's clarify that it is possible to launch Puppeteer and Playwright with other executables than the bundled one in the case of Chrome/Chromium, so it is not specific to Cypress.

    const browser = await puppeteer.launch({ executablePath: '/path/to/Chrome70' });

    const browser = await chromium.launch({ executablePath: '/path/to/Chrome70' });

    Second, even if 'testing' is there in the list of features in the case of Puppeteer and Playwright, they are not testing frameworks while Cypress is a testing tool primarily. Update (2023): Since I wrote this answer I'd argue with my statement that Playwright is not primarily a testing framework. It is indeed primarily a testing framework, see @playwright/test is more popular than the regular playwright library itself.

    What's the benefit of E2E testing only against the latest browser versions?

    Some years ago it was crucial to test across all browsers and as many versions as used by a mass of users.
    Over time it has changed.

    What's new in 2021?

    • Only Google Chrome and Safari (Webkit) browsers have 10%+ market share. Statcounter.com, Wikpedia
    • For the vast majority not even Firefox support is required (with its 3.68% market share worldwide)
    • "Google Chrome on Windows and Mac auto-updates itself on a regular basis. The auto-updating procedure is performed by Google Update, which is based on the open-source Omaha project. Auto-updated provides fixes to sometimes critical issues, limiting exposure." chromium.org
    • Due to the auto-updates the number of users with not the latest Chrome is very marginal. The previous versions (e.g. the last-but-one) go out of circulation in 1-2 months and remains under 1% share for a short time before vanishing. Desktop Browser Version Market Share Worldwide, Statcounter.com

    Desktop Browser Version Market Share Worldwide

    Conclusion(s)

    I.) It means for most products it is fairly enough and the best option to test against the latest browser versions.

    You've mentioned Chrome 70 which is a relatively old version, released in October 2018, for most products it is not required to support it, they are in the same bucket as Internet Explorer 11 or legacy Edge.

    II.) For some products, you may have to support specific (older) browser versions, in that case, the usage of specific executables can help.

    By the way, if you decide to run tests against multiple browser versions it will be still possible with Puppeteer or Playwright as well, you just need to provide the right executables while the test suites are iterated over (e.g. in case of Jest: describe.each() can run the same tests or test suites with different configs and test data).