Tornado server does not receive big files

Question

Hi I'm trying to realize a Tornado server with the goal to receive very big binary files (~1GB) into POST body. The following code works for small files, but does not answer if I try to send big files (~100MB).

class ReceiveLogs(tornado.web.RequestHandler):  
   def post(self):    
      file1 = self.request.body
      output_file = open('./output.zip', 'wb')
      output_file.write(file1)
      output_file.close()        
      self.finish("file is uploaded")

Do you know any solutions?

Answer 1

I don't have a real implementation as an answer but one or two remarks that hopefully point to the right direction.

First of all there is a 100MB Upload limit which can be increased setting the

self.request.connection.set_max_body_size(size) 

in the initalization of the Request handler. (taken from this answer)

The Problem is that tornado handles all file uploads in memory (and that HTTP is not a very reliable Protocol for handling large file uploads.) This is quote from a member of the tornadoweb team from 2014 (see github issue here)

... You can adjust this limit with the max_buffer_size argument to the HTTPServer constructor, although I don't think it would be a good idea to set this larger than say 100MB.

Tornado does not currently support very large file uploads. Better support is coming (#1021) and the nginx upload module is a popular workaround in the meantime. However, I would advise against doing 1GB+ uploads in a single HTTP POST in any case, because HTTP alone does not have good support for resuming a partially-completed upload (in addition to the aforementioned error problem). Consider a multi-step upload process like Dropbox's chunked_upload and commit_chunked_upload (https://www.dropbox.com/developers/core/docs#chunked-upload)

As stated I would recommend to do one of the following:

• if NGNIX is possible to handle and route requests to tornado=> look at the NGNIX upload module (see ngnix wiki here)
• If it must be a plain tornado solution use the tornado.web.stream_request_body which came with tornado 4. This streams the uploaded files to disk instead of trying to first get them all in mem. (see tornado 4 release notes and this solution on stackoverflow)