POSTing and GETing with grequests

Question

I'm using grequests to scape websites faster. However, I also need to login to the website.

Before (just using requests) I could do:

where headers is my User-Agent.

with requests.Session() as s: 
    s.headers.update(headers)
    s.post(loginURL, files = data)
    s.get(scrapeURL)

Using grequests I've only been able to pass headers by doing:

rs = (grequests.get(u, headers=header) for u in urls)
response = grequests.map(rs)

Is there anyway to do a POST at the same time so I can login? The login URL is differnt than the URL(s) I'm scrapping.

Answer 1

You can pass in the Session object exactly the same as the headers:

with requests.Session() as s: 
    s.headers.update(headers)
    s.post(loginURL, files = data)
    s.get(scrapeURL)


    rs = (grequests.get(u, headers=header, session=s) for u in urls)
    response = grequests.map(rs)