I can't understand why with this code the requests.get()
returns immediately error and the 10 second timeout is not respected.
If I remove the header parameter, so only requests.get(url, proxies={"http": proxy, "https": proxy}, verify=False,timeout=10)
, the timeout is respected and a request is returned successfully or failure if the timeout occurs.
I need to insert headers and a timeout too to test proxies, how to do it?
from fake_useragent import UserAgent
import requests
ua = UserAgent()
#get a list of proxy
for i in range(1, len(proxies)):
# Get a proxy from the pool
proxy = next(proxy_pool)
print("Request #%d" % i)
try:
response = requests.get(url, headers=ua.random(), proxies={"http": proxy, "https": proxy}, verify=False,timeout=10)
break
except:
print("Skipping. Connnection error")
try with this
ua = UserAgent()
header = {'User-Agent':str(ua.random)}
response = requests.get(url, headers=header, verify=False,timeout=10)