I use selenium and bs4 to parse web page. And the web page use scan qrcode and verification code to login.
I use WebDriverWait
to wait the user to login with
wait = ui.WebDriverWait(driver, 60) # timeout after 60 seconds, just leave time for user to login
wait.until(lambda driver: driver.execute_script('return isLogin();'))
and after I login, then begin the parsing work.
It's works well, except that each time when I run the script, it open a new broswr window and I need to login.
How to retain the login state to avoid login when each time run the script.
To avoid login every time you can use cookies. For example:
import pickle
from selenium import webdriver
browser = webdriver.Chrome()
browser.get("http://www.google.com")
pickle.dump(browser.get_cookies(), open("cookies.pkl","wb"))
and download cookies later:
import pickle
from selenium import webdriver
browser = webdriver.Chrome()
browser.get("http://www.google.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
browser.add_cookie(cookie)