I am trying to scroll down a YouTube Channel's "Videos" tab so I can obtain the links to all the videos in that channel. I am using the Selenium module in Python 3, with Google Chrome as the browser. This is the method in my class that does this part (this is called right after I call self.driver.get(CHANNEL_URL))
def get_video_links(self):
xpath = '/html/body/ytd-app/div/ytd-page-manager/ytd-browse/ytd-two-column-browse-results-renderer/div[1]/ytd-section-list-renderer/div[2]/ytd-item-section-renderer/div[3]/ytd-grid-renderer/div[1]/ytd-grid-video-renderer/div[1]/div[1]/div[1]/h3/a'
elements = self.driver.find_elements_by_xpath(xpath)
last_result = 0
curr_result = len(elements)
while curr_result != last_result:
self.driver.find_element_by_tag_name('body').send_keys(Keys.END)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(3)
elements = self.driver.find_elements_by_xpath(xpath)
last_result = curr_result
curr_result = len(elements)
video_links = [elem.get_attribute('href') for elem in elements]
return video_links
This currently returns 60 urls, while doing no scrolling gets me 30 urls. The channel I'm scraping, however, has around 150 videos. I have succeeded with manual scrolling, so I know the element-finding part works. I have tried this and this but they only get me 30 urls and I don't see any scrolling happening on the browser.
You can use JavaScript to scroll down in selenium.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Taken from the selenium docs.
If you just want to download all the videos a channel has made, youtube-dl is the better job.
All you have to do is run youtube-dl from the console and add the Channel ID for the channel you want downloaded.
$ youtube-dl "[INSERT CHANNEL-ID]"
It also works on playlists and individual videos and even supports other sites.