Search code examples
pythonselenium-webdriverselenium-chromedriver

Bugs after targeting dynamically generated web elements using XPATH statements

As the subject mentions I'm having trouble when targeting a web element using XPATH statements. Upon initial startup of my program, the script will work for a hundreds of entries using information retrieved from a separate file. The problem arises when a seeming change of structure within the DOM which causes the bug to occur. Causing selenium's WebDriver to retarget an unwanted element previously referenced in the program.

Video of the bug

To give more information, the element is being targeted by XPATH through this code snippet.


    def diagnosis(self, driver, patient):
        try:
            wait = WebDriverWait(driver,10)
            valid_diagnoses = [dx for dx in patient.diagnosis if dx and len(str(dx)) < 1000 and not re.search(r"\bnan\b",str(dx))]
            xpath_id = [7, 10, 13, 16, 19, 22, 25]
            count = 0
            diagnosis_index = 0

            while count < len(valid_diagnoses)/2:
                wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-sm.btn-secondary.font-medium.text-sm.text-gray-700"))).click()
                time.sleep(2)
                try:
                    diagnosis_field = None
                    diagnosis_field = wait.until(EC.visibility_of_element_located((By.XPATH, f'//*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[{xpath_id[count]}]//input[not(@disabled)]')))
                    text = valid_diagnoses[diagnosis_index].strip()
                    diagnosis_field.send_keys(text)
                    wait.until(EC.text_to_be_present_in_element_value((By.XPATH, f'//*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[{xpath_id[count]}]//input[not(@disabled)]'), text))
                except:
                    print(f'An error related to the patient\'s diagnosis being entered is occurring. Program will wait 100 seconds, inspect the HTML')
                    time.sleep(100)
                    patient.dx_issue = True
                diagnosis_index += 2
                count += 1
        except TimeoutError:
            print("Timed out while completing the diagnosis process in Webdriver.py")

The elements on the DOM being targeted Diagnosis 1's DOM representation The image above's XPATH when copied from the inspector: //*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[10]/div[2]/input

Diagnosis 2's DOM representation The image above's XPATH when copied from the inspector: //*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[10]/div[2]/input

To give more context

I've used different types of selectors, iterating through different CSS_Selectors and XPATH statements, XPATHs were ultimately best for my requirements and tackled the problem by first using //*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[{xpath_id[count]}]/div[1]/input then by modifying it into //*[@id="919e06c4ea7e2a5bb720134d693a8671"]/div[2]/div[1]/div[2]/div/form/div/div/div[{xpath_id[count]}]//input[not(@disabled)]

I found that the second method more reliably targets the WebElement because there could be possible DOM changes that I did not account for with the first statement. This method has proved to be more resilient and tailored to my requirements as it has been working with hundreds of entries.

The possible problem I may have overlooked was Selenium's limitations when it comes to running the WebDriver for an extended period of time. I'm relatively new to selenium so I don't fully understand the limits of how long the program can continue without eventually running into bugs. Especially with hundreds of entries each with details of a unique office visit.

REFERENCES

If anyone can provide more insight into helping me solve my problem it would be much appreciated. Thank you!

ADDITIONAL INFORMATION

  • Google Chrome Version 122.0.6261.95 (Official Build) (64-bit)
  • Windows 11 Home
  • selenium 4.15.2

EDIT

Included HTML page for greater insight into the structure of the DOM

Solution

  • Below is the relevant HTML for the INPUT you are looking for. I've clipped some elements and attributes out that aren't relevant to make it easier to read.

    <div class="col-span-6">
    <label class="..."> Diagnosis 2: </label>
    ...
    <div>
        <input class="..." wire:model.debounce.500ms="search_diagnoses.1" type="text" placeholder="Search diagnosis...">
        ...
    </div>
</div>

    The locator below uniquely identifies the INPUT above,

    //label[contains(text(),'Diagnosis 2:')]/following-sibling::div/input[@placeholder='Search diagnosis...']
    Syntax Meaning
    //label[contains(text(),'Diagnosis 2:')] A LABEL tag that contains the text, "Diagnosis 2:"
    /following-sibling::div A DIV tag that is a sibling of the LABEL above
    /input[@placeholder='Search diagnosis...'] A child INPUT of the LABEL above that contains the attribute placeholder that contains the text, "Search diagnosis..."