Search code examples
pythonpython-3.xseleniuminfinite-scrollmobile-website

Is there a way to scroll down or trigger infinite scroll loading on mobile web site?

I've been searching for the way to scroll down mobile web site. This is the url example I want to crawl:

https://m.place.naver.com/place/list?query=%EC%84%B1%EC%88%98%EB%8F%99%20%EA%B7%BC%EC%B2%98%20%EB%8C%80%ED%98%95%EB%A7%88%ED%8A%B8&x=127.04965&y=37.542108&start=501&display=300&adult=true&spq=false&highlight=true&sessionId=bch8Eyf4fKKe0ppVJkHJKg%3D%3D&sortingOrder=distance&level=top

As you can see, it doesn't have a scroll bar in it, and it only shows 50 results for initial loading. I tried

y = driver.find_elements_by_xpath("//li[@class='_387oy']")
print(len(y))
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
print(len(y))

But it doesn't work. It shows 50 results before and after execute_script.

As you may already assume, this url does have a pc version but it doesn't have a data which I need. Only in mobile version, I can set the GPS coordinates in url parameter, and so it calculate and brings me back the results with the distance from the coordinates. So... I think I should work on this mobile site.

But as I already tell you, I can't scroll down all the data. And I'm not good at JS, so I'm not sure if there is a work around with other execute_script which can allow trigger loading more data as you scroll down to bottom, or just display all the data at once.

FYI, among url parameter, it does have a parameter of "display" and "start" but I don't know how they function. When I changed display from 30 to 300, or 30000, no changes for a result.

It would be very helpful if anyone can share some knowledge of workaround. Thank you in advance!

Solution

  • You need scroll to the last element with this argument : arguments[0].scrollIntoView(true);

    Use [-1] to get last element in your list.

    And after scroll, initialize again your element to get new size.

    y = driver.find_elements_by_xpath("//li[@class='_387oy']")
print(len(y))
last_el = y[-1]
driver.execute_script("arguments[0].scrollIntoView(true);", last_el)
time.sleep(5)
y = driver.find_elements_by_xpath("//li[@class='_387oy']")
print(len(y))

    The best way is to use a loop.