Getting "StaleElementReferenceException" when accessing options in drop down menu

Question

In Python and Selenium, I'm populating a form, submitting it, then scraping the resulting multi-page table that appears on the page underneath the form. After I scrape every page of this table, I reset the form and attempt to repopulate the form. However, a drop down menu is tripping up the code.

I've tried to make the driver wait for the drop down menu to reappear after I reset the form, but this doesn't help. I still receive the StaleReferenceElementException error on the if option.text == state line:

 StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

How do I submit the form over and over for different options within the drop down menu?

states = ['Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California', 
          'Colorado', 'Connecticut', 'Delaware', 'District of Columbia', 
          'Florida', 'Georgia', 'Hawaii', 'Idaho', 'Illinois', 'Indiana',
          'Iowa', 'Kansas', 'Kentucky', 'Louisiana', 'Maine', 'Maryland',
          'Massachusetts', 'Michigan', 'Minnesota', 'Mississippi', 'Missouri',
          'Montana', 'Nebraska', 'Nevada', 'New Hampshire', 'New Jersey', 
          'New Mexico', 'New York', 'North Carolina', 'North Dakota', 'Ohio',
          'Oklahoma', 'Oregon', 'Pennsylvania', 'Rhode Island', 'South Carolina',
          'South Dakota', 'Tennessee', 'Texas', 'Utah', 'Vermont', 'Virginia',
          'Washington', 'West Virginia', 'Wisconsin', 'Wyoming']

# Construct browser and link
browser = webdriver.Firefox(executable_path='/usr/local/bin/geckodriver')
url = 'https://myaccount.rid.org/Public/Search/Member.aspx'
ignored_exceptions = (StaleElementReferenceException,)

# Navigate to link
browser.get(url) 

try:
    # For each state
    for state in states:
        print('Searching ' + state)
        # Set category and select state menu
        category = Select(browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$categoryDropDownList'))
        category.select_by_value('a027b6c0-07bb-4301-b9b5-1b38dcdc59b6')
        state_menu = Select(WebDriverWait(browser, 10, ignored_exceptions=ignored_exceptions).until(EC.presence_of_element_located((By.ID, 'FormContentPlaceHolder_Panel_stateDropDownList'))))
        options = state_menu.options

        for option in options:
            if option.text == state:
                state_menu.select_by_value(option.get_attribute('value'))
                browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$searchButton').click()

                # Scrape the first page of results
                results = []
                curr_page = 1
                onFirstPage = True
                scrape_page(curr_page)

                # Reset form
                browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$resetButton').click()
                break
finally:
    pass

Answer 1

The moment you select the option, element references will update and you can't use the older references. Reason you are getting the exception is, you are trying to get the attribute from the option which no longer valid.

Rather using the iteration, I would use the xpath to select the option as shown below

        state_menu = WebDriverWait(browser, 10, ignored_exceptions=ignored_exceptions).until(EC.presence_of_element_located((By.ID, 'FormContentPlaceHolder_Panel_stateDropDownList')))
        #options = state_menu.options <== replace this line with below line
        option = state_menu.find_element_by_xpath("//option[.='" + state + "']")

        #for option in options: <== remove this line
            # if option.text == state: <== remove this
        option.click()
        browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$searchButton').click()

        # Scrape the first page of results
        results = []
        curr_page = 1
        onFirstPage = True
        scrape_page(curr_page)
        # Reset form
        browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$resetButton').click()