The crypto exchange hotbit.io disabled some important endpoints I was using. Now I have to use the Cloudflare protected ones. My Idea was to solve Cloudflare with selenium/undetected_chromedriver and pass the cookies to my session.
Code:
import time
import requests
import undetected_chromedriver as uc
#create a browser and solve cloudflare
driver = uc.Chrome()
url = "https://www.hotbit.io/"
driver.get(url)
time.sleep(5)
#storing the cookies generated by the browser
request_cookies_browser = driver.get_cookies()
print(request_cookies_browser)
s = requests.Session()
#passing the cookies generated from the browser to the session
c = [s.cookies.set(c['name'], c['value']) for c in request_cookies_browser]
print(s.cookies)
resp = s.get(url)
print(resp.text)
I changed some values so I'm not leaking anything
Output: Cookies: https://pastebin.com/u7sGvjae html-hotbit: https://pastebin.com/6sAfhWtv
It looks like all cookies are set correctly but I'm still on the Cloudflare solving page with my session. Anyone has a solution to "bypass" the Cloudflare page and access the website with requests?
Looks like cookies AND user-agent are needed. To be precise only the cf_clearance cookie AND user-agent are needed.
Solution would looks something like that:
import time
import undetected_chromedriver as uc
import requests
#create driver with selenium/undetected_chromedriver
driver = uc.Chrome()
driver.get("https://www.hotbit.io/")
time.sleep(5) #5s sleep, so the driver can solve cloudflare
#get cookies and user_agent
brCookies = driver.get_cookies()
ua = driver.execute_script("return navigator.userAgent")
#filter out the cf_clearance cookie
cf_cookies = [cookie for cookie in brCookies if cookie['name'] == 'cf_clearance'][0]['value']
cookies = {"cf_clearance": cf_cookies}
headers = {"user-agent": ua}
#create session and get your site
session = requests.session()
res = session.get("https://www.hotbit.io/", headers=headers, cookies=cookies)