Running Python script containing selenium from Azure Pipelines. Not able to locate elements

Question

I have a Python script which performs selenium tasks. The script is run through Azure pipelines with the following yaml setup

# Python package
# Create and test a Python package on multiple Python versions. OK
# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/python

trigger: none

schedules:
- cron: "0/5 * * * 4"
  displayName: Run every couple of minutes
  branches:
    include:
    - develop

pool:
  vmImage: 'ubuntu-latest'
  name: Azure Pipelines

variables:
- group: Secrets&Passwords


stages:
- stage: Deployment
  displayName: Deployment stage
  jobs:
    - deployment: deploymentJob
      displayName: Deploy
      environment:
       name: Test
       resourceType: VirtualMachine
      strategy:
       runOnce:
         deploy:
           steps:
           - task: CmdLine@2
             inputs:
               script: |
                 python -m pip install --upgrade pip 
                 python -m pip install selenium 
                 python -m pip install pdfplumber 
                 python -m pip install pandas
             displayName: 'Install dependencies'
           - task: PythonScript@0
             displayName: 'Run a Python script'
             inputs:
              scriptPath: 'C:/azagent/A2/_work/Manuelle Ordrer/dist/main.py'
              arguments: -myvariable1 $(myvariable1) -myvariable2 $(myvariable2) -myvariable3 $(myvariable3) 

I have indicated the selenium to run as headless with options.headless = True since the following setup runs in the background. When I manually start the python job on my local machine it works fine, however when it is run through Azure Pipelines, selenium is not able to locate the first element and I get the error

"Unable to locate element: {"method":"xpath","selector":"//*[@id="user"]"} (Session info: headless chrome=92.0.4515.159)"

Does anybody have an idea why this is only the case when running it from Azure Pipelines?

Answer 1

"Unable to locate element: {"method":"xpath","selector":"//*[@id="user"]"} (Session info: headless chrome=92.0.4515.159)"

It could be that when the selenium test is run in the Pipeline, the web page loading speed will be slower than the local loading.

So it can cause this issue.

You can try to add the waiting time in your python script.

For example:

driver = webdriver.Chrome()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()