I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this API request can have 500+ items, so finishing this looping is taking 20+ minutes sometimes.
I tried to use the grequests library and I kept getting recursion complaints. I would love to use an async/map method if possible but after researching apparently the async library is not supported anymore.
self.set_header("Access-Control-Allow-Origin", "*")
response = requests.get("https://hacker-news.firebaseio.com/v0/paststories.json?print=pretty")
data = response.json()
story_list = []
for story in data:
temp_string = "https://hacker-news.firebaseio.com/v0/item/{}.json?print=pretty".format(story)
story_data = requests.get(temp_string)
story_list.append(story_data.json())
There should be a better way to do this looping than the current method because 20+ minutes to get the data is not acceptable. The API response in the original array can return an array of 500+ so the method should be scalable.
requests is synchronous, so your script waits for a response to make a new request. So Maybe you should look into aiohttp and aysnchronous requests.
This could be an example : Is that benchmark reliable - aiohttp vs requests