Search code examples
pythonpython-2.7for-loopscrapy

Get to load "more" button for next page?

I am trying to click the "more" button for getting next set of pages.

Next page link Htmltags:

<span id="direct_moreLessLinks_listingDiv" class="more_less_links_container" data-num-items="15" offset="45" data-type="listing">
<a id="button_moreJobs" rel="nofollow" href="#">More</a>
</span>

Note: when I am clicking the "more" button. It is loading the next page in the same page.

I am trying to click the next page using selenium:

Code part:

self.driver.get('example.com')
        while True:
            more_btn = WebDriverWait(self.driver, 10).until(
                EC.visibility_of_element_located((By.ID, "More"))
            )

            More.click()

            # stop when we reach the desired page

Solution

  • Find the link by ID and click it:

    more_btn = WebDriverWait(self.driver, 10).until(
               EC.visibility_of_element_located((By.ID, "button_moreJobs"))
           )
more_btn.click()

    Since this is a while True loop, you need to figure out a way to stop the pagination and break the loop. This is something I can only make guesses about since you haven't provided an actual page you are working with.