How to read data from web socket before socket completion

Question

I'm trying to build client-server app in Python. My client use requests module to connect to the server and upload json and files.

Server use tornado framework. When server receive data from client, they start processing and send result to client by parts. Example of my post handler:

class PostAd(tornado.web.RequestHandler):

def post(self):
    jdata = self.get_body_arguments('json', False)[0]
    jdata = json.loads(jdata)
    id = self.insert_ad(jdata)
    fpath_list = self.save_files(self.request.files.values(), id)
    self.insert_file_path(id, fpath_list)
    self.write("Successfully posted into SQL with sql id: {0}".format(id))
    self.flush()
    self.write("Are u there?")
    self.finish()

in the client requests used to post data

r=agent.post("http://localhost:8888/api/v1/add-ad", data={"json": thread_data}, files=files)

in this way I cannot receive data by pieces because r=agent.post will wait until server will close connection but I need to check returned values every time when tornado server will send me data with self.flush() command (in my example I expect to get two answers, first one: "Successfully posted into SQL with sql id: 100" and second: "Are u there?").

is it possible to do it with requests module or I need to use something else here?

Answer 1

I don't know what agent.post() is, but you can do this with tornado's HTTP client and the streaming_callback option. You'll have to format the request body yourself, though, since Tornado doesn't have built-in client-side support for multipart file uploads.

await AsyncHTTPClient().fetch(url, body=encoded_body, streaming_callback=print)

There is no guarantee that the chunks observed by streaming_callback will align with the calls to flush, so you should format the data so that the client can determine where messages begin or end.