Search code examples
pythonseleniumweb-scrapingdata-miningdata-extraction

Selenium unable to locate locate element within a fieldset


I wanted to scrape data from a website which is "https://fhsaa.com/sports/2020/1/28/member_directory.aspx". The idea is to search school names using id or school name and then scrape the information. but when I used xpath or css_selecotr there is an exception "o such element: Unable to locate element: {"method":"xpath","selector":"//input[@name='school_name']"}

" I have used both xpath, css_selector and waits too but no success. Here is the code...

<fieldset class="border">
            <legend class="border">School Filter</legend>
            <form method="GET" action="https://fhsaahome.org/widget/school-directory/submit" accept-charset="UTF-8">
                <div class="row">
                    
                    <div class="col-md-4">
                        
                        <select class="form-control" name="school_id"><option selected="selected" value="">--Select--</option><option value="714">Abundant Life Christian (Margate)</option><option value="665">Academy at the Lakes (Land O'Lakes)</option><option value="639">Academy Prep (St. Petersburg)</option><option value="4">Admiral Farragut (St. Petersburg)</option><option value="1191">AIE Charter (Miami Springs)</option>value="542">Zephyrhills</option><option value="1062">Zephyrhills Christian</option></select>
                    </div>
                    <div class="col-md-4">
                        
                        <input class="form-control" placeholder="Enter School Name" name="school_name" type="text" value="">
                    </div>
                    <br><br>
                    <div class="col-md-2">
                        <input class="btn btn-secondary btn-md" type="submit" value="Search">
                      
                    </div>
                
                </div> 

Solution

  • This is an iframe game, I see the website has called an API to continuously shuffle the xpath you are looking. I add a piece of code to switch between iframes and they worked:

    iframe = driver.find_elements_by_tag_name('iframe')[0]
    driver.switch_to.frame(iframe)
    element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')

    This is the full program:

    import os
    from selenium import webdriver
    
    from time import sleep
    from random import randint
    
    chromedriver = "C:\Program Files\Python39\Scripts\chromedriver"
    os.environ["webdriver.chrome.driver"] = chromedriver
    driver = webdriver.Chrome(chromedriver)
    url = 'https://fhsaa.com/sports/2020/1/28/member_directory.aspx'
    driver.get(url)
    sleep(randint(2,3))
    
    iframe = driver.find_elements_by_tag_name('iframe')[0]
    driver.switch_to.frame(iframe)
    element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
    
    element.send_keys('yourschool')
    sleep(randint(2,3))