Search code examples
htmlpython-3.xrequestgrequests

Extracting Text From Asynchronus Request Using grequests

I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so.

If we use Requests Library I would do

r = requests.get('www.google.com')
htmls.append(r.text)

Now if I am using grequests I can only get a list of response code and not text.

rs = (grequests.get(u) for u in urls)
result = grequests.map(rs)

What I've tried result = grequests.map(rs.text)

I get an error using above piece of code AttributeError: 'generator' object has no attribute 'text'

My desired output is a list of html text where response code is 200 else the value should be None. How can I achieve that?

Desired Output:

response_code = [<Response [200]>,<Response [404]>,<Response [200]>]
htmls = ['html1', None, 'html2']
Solution

  • You can use something like below

    rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)
text = list(map(lambda d : d.text if d else None, responses))

print(text)

    What you are getting back is a response array after you call the map. And then you can process this data using native map function