python tornado get response of multiple async httprequest

Question

i have a list of url_handler and i would want to make asyncronous httprequest using tornado. When all response structure is arrived i need to use it for other targets. Here a simple example of my code:

(...)
self.number = 0
self.counter = 0
self.data = {}
(...)
@tornado.web.asynchronous
def post(self):
    list_url = [url_service1, url_service2]
    self.number = len(list_url)
    http_client = AsyncHTTPClient()
    for service in list_url:
        request = tornado.httpclient.HTTPRequest(url=service, method='POST', headers={'content-type': 'application/json'}, body=json.dumps({..params..}))
        http_client.fetch(request, callback=self.handle_response)

    # Loop for is finished. Use self.data for example in other funcions...
    # if i print(self.data) i have empty dict...
    # do_something(self.data)



def handle_response(self,response):
    if response.error:
        print("Error")
    else:
        self.counter = self.counter + 1
        print("Response {} / {} from {}".format(self.counter, self.number, response.effective_url))
        self.data[response.effective_url] = json_decode(response.body)
    # number is 2
    if self.counter == self.number:
        print("Finish response")

def do_something(data):
    # code with data parameter

I hope my problem is well explained

Answer 1

Since you know AsyncHTTPClient is asynchronous, that means, the requests will run in background.

So, when the for loop is finished, that does not mean all the requests are also finished - they are running in the background even when the loop finishes.

That is why self.data is empty, because the requests aren't completed yet.

---

How to fix this

As you know the handle_response callback is called after every request is completed. You can call do_something function from this callback when all the requests are completed. Like this:

def handle_response(...):
    ...
    if self.counter == self.number:
        self.do_something(self.data)
        print("Finish response")