Web scraping for Linkedin

Question

I am currently working on a college project for Linkedin Web Scraping using selenium. Following is the code for the same:

from selenium import webdriver
from time import sleep
from selenium.webdriver.common.keys import Keys
from parsel import Selector

driver = webdriver.Chrome('location of web driver')
driver.get('https://www.linkedin.com')

# username
username = driver.find_element_by_id('session_key')
username.send_keys('Linkedin Username')
sleep(0.5)

# password
password = driver.find_element_by_id('session_password')
password.send_keys('Linkedin Password')
sleep(0.5)

#submit value
sign_in_button = driver.find_element_by_xpath('//*[@type="submit"]')
sign_in_button.click()
sleep(0.5)

driver.get('https://www.google.com/')   #Navigate to google to search the profile

# locate search form by_name
search_query = driver.find_element_by_name('q')

# send_keys() to simulate the search text key strokes
search_query.send_keys('https://www.linkedin.com/in/khushi-thakkar-906b56188/')
sleep(0.5)

search_query.send_keys(Keys.RETURN)
sleep(3)

# locate the first link
search_person = driver.find_element_by_class_name('yuRUbf')
search_person.click()

#Experience
experience = driver.find_elements_by_css_selector('#experience-section .pv-profile-section')
for item in experience:
    print(item.text)
    print("")

#Education
education = driver.find_elements_by_css_selector('#education-section .pv-profile-section')
for item in education:
    print(item.text)
    print("")

#Certification
certification = driver.find_elements_by_css_selector('#certifications-section .pv-profile-section')
for item in certification:
    print(item.text)
    print("")

When I scrape the experience part, it extracts the information perfectly. But when I do the same with Education and certifications part - It shows an empty list. Please help!

Answer 1

I think the problem ís because of your css selector. I try it my self and it is unable to locate any element on html main body

Fix your css selector and you will be fine

#Education
education = driver.find_elements_by_css_selector('#education-section li')

#Certification
certification = driver.find_elements_by_css_selector('#certifications-section li')