Search code examples
seleniumgoogle-chromejenkinsselenium-chromedriverrenderer

Selenium build unstable in Jenkins but fine locally

SOLUTIONS POSTED BELOW - TLDR; this dumbs down to using presence_of_element_located with the wrong use case and then not thinking of browser and driver desync.

ISSUE: Issue in which tests pass locally but fail via Jenkins.

Local always passes (windows VM)

Local

Jenkins sometimes passes (windows VM)

Jenkins

Both the local tests and Jenkins tests are run on the same VM (Windows agent). I have followed the advice from these people by adding the following arguments with no change/success:

options = Options()
options.add_experimental_option('excludeSwitches', ['enable-logging'])  #Fixes the problem with console outputing warning "Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed."
options.add_argument('--no-sandbox')
options.add_argument("enable-automation")
#options.add_argument("--headless") #CAUSES TIMEOUT ERROR
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-extensions")
options.add_argument("--dns-prefetch-disable")
options.add_argument("--disable-gpu")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-browser-side-navigation")
options.add_argument("--force-device-scale-factor=1")
prefs = {"download.default_directory" : r"C:\Users\MyUser\Path\Directory\\"} 
options.add_experimental_option("prefs", prefs)
s = Service('chromeDriverPath/chromedriver.exe')
context.driver = webdriver.Chrome(service=s, options=options,desired_capabilities=capa)

The --headless option creates an error:

Timed out receiving message from renderer: 10.000

so I have commented that out.

Question: Why does my local always pass but my Jenkins is unstable even though they run tests on the same VM. How can I fix this?

Background

I have looked at this similar question but no solution from it. Unfortunately I have spent many hours trying to fix this.

  • Not the code - Im positive its not the code because I did all the code myself (Python + behave). Just meaning I tested it all through its creation extensively and once again it passes every time locally in Dev.
  • Driver and Browser sync - Im looking into "the sync between the Selenium driver and the browser is the issue" as stated here by @ekostadinov.

Installed Package Versions (PIP)

Package               Version
--------------------- ---------
allure-behave         2.9.45
allure-python-commons 2.9.45
async-generator       1.10
attrs                 21.4.0
behave                1.2.6
cairocffi             1.3.0
CairoSVG              2.5.2
certifi               2022.6.15
cffi                  1.15.1
charset-normalizer    2.1.0
chromedriver          2.24.1
click                 8.1.3
colorama              0.4.5
cryptography          37.0.4
cssselect2            0.6.0
defusedxml            0.7.1
et-xmlfile            1.1.0
h11                   0.13.0
idna                  3.3
lxml                  4.9.1
multipledispatch      0.6.0
numpy                 1.23.1
openpyxl              3.0.10
outcome               1.2.0
pandas                1.4.3
parse                 1.19.0
parse-type            0.6.0
Pillow                9.2.0
pip                   22.0.4
pluggy                1.0.0
pycparser             2.21
pygal                 3.0.0
pyOpenSSL             22.0.0
pypiwin32             223
pyshadow              0.0.4
PySocks               1.7.1
python-dateutil       2.8.2
python-docx           0.8.11
python-dotenv         0.20.0
pytz                  2022.1
pywin32               304
requests              2.28.1
selenium              4.3.0
setuptools            58.1.0
six                   1.16.0
sniffio               1.2.0
sortedcontainers      2.4.0
tinycss2              1.1.1
trio                  0.21.0
trio-websocket        0.9.2
urllib3               1.26.10
webdriver-manager     3.8.1
webencodings          0.5.1
wsproto               1.1.0

Update 08/02/2022 4:11pm

Adding more implicit and explicit wait time fixed many of the flaky tests

  • Some places needed time.sleep(x) to wait for downloads to finish before extracting csv/xlsx the timeout time bumped up on WebDriverWait(context.driver, timeOutTimeInSeconds).until(EC.presence_of_element_located((By.CSS_SELECTOR, '{$path}')))

Still facing some flaky tests only through Jenkins involving WebDriverWait(context.driver, 180).until(EC.presence_of_element_located((By.CSS_SELECTOR, '{$path}')))

Update 08/03/2022 8:20am SOLUTION

  1. As pointed out by @undected Selenium using presence_of_element_located for this use case may be incorrect. It seems to be the source of the flaky tests. Alternative implementation can be found via the docs. This explains more about why it is bad practice for some scenarios:
  • "If your usecase is to validate the presence of any element you need to induce WebDriverWait setting the expected_conditions as presence_of_element_located() which is the expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible" (@undetected Selenium). (Source)

One issue was I needed to wait until an element becomes visible instead of just in the DOM. I had to change some presencee_of_element_located() to visibility_of_element_located():

  • "If your usecase is to extract any attribute of any element you need to induce WebDriverWait setting the expected_conditions as visibility_of_element_located(locator) which is an expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0." (@undetected Selenium). (Source)
  1. Adding extra time to explicit wait times fixed many flaky tests that were caused from the desync between browser and driver.

Example

Before

WebDriverWait(context.driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '{$path}')))

After

WebDriverWait(context.driver, 60).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '{$path}')))

Summary

Using the correct expected condition is important. I even had to alter one of mine to element_to_be_clickable to make it not flaky. It was a test involving waiting for a clickable element to pop up anyway. Make sure to read the docs and use the best fit expected condition to avoid errors. Also, look into extending implicit or explicit wait times if its desync errors. Screenshotting upon fail can help you debug this.

Solution

  • There are couple of things you need to address as follows:

    • Presumably you are on a system, so you need to drop the argument:

      options.add_argument("--disable-gpu")

      as the argument --disable-gpu was effective only for non-windows environment and is no more relevant.

    • Some of the arguments are no more relevant, so you can drop them:

      options.add_argument("enable-automation")
options.add_argument("--disable-extensions")

    • Ideally test should be executed as a non-root / non-admin user, so you may not need the arguments:

      options.add_argument('--no-sandbox')
options.add_argument("--disable-dev-shm-usage")

    • Possibly you don't require the either of the following arguments as well:

      options.add_argument("--dns-prefetch-disable")
options.add_argument("--disable-browser-side-navigation")
options.add_argument("--force-device-scale-factor=1")

    • You may like to tweak the other argument as:

      prefs = {"download.default_directory" : r'C:\Users\MyUser\Path\Directory\'} 
options.add_experimental_option("prefs", prefs)

    • Finally, you aren't using any capabilities so you can remove the argument from the constructor:

      desired_capabilities=capa

    Solution

    Your effective code block can be:

    options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option("excludeSwitches", ["enable-automation"])
prefs = {"download.default_directory" : r'C:\Users\MyUser\Path\Directory\'} 
options.add_experimental_option("prefs", prefs)
s = Service('chromeDriverPath/chromedriver.exe')
context.driver = webdriver.Chrome(service=s, options=options)