Search code examples
pythonpython-requestspyramid

Requests and Requests_Toolbelt filesize limit


Using request_toolbelt's MultipartEncoder, I am able to upload files to my Pyramid server, but only up to a certain size. Once the file are beyond a certain size (not sure of the exact size), requests throws an ConnectionError. My Pyramid server shows absolutely no sign of ever getting the request. The callback monitor shows 2 or 3 chunks read but then it aborts. Why?? Here is my code:

### client-side

# callback
def cb(monitor):
    print monitor.bytes_read

file = open('my_big_file.mpg', 'rb')
payload = MultipartEncoder({'uploadedFile': (file.name, file, 'application/octet-stream')})
monitor = MultipartEncoderMonitor(payload, cb)
r = requests.post(url, data=monitor, headers={'Content-Type': payload.content_type})

#### server-side

@view_config(route_name='remote.agent_upload', renderer='json')
def remote_agent_upload(request):
r = request.response
uploadedFile = request.POST['uploadedFile']
fs = uploadedFile.file
filename = uploadedFile.filename
f = open('path_to_storage' + filename, 'wb')
f.write(fs.read())
fs.close()
f.close()
return r

### output and traceback

8192
16384
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\site-packages\requests\api.py", line 109, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "c:\Python27\lib\site-packages\requests\api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "c:\Python27\lib\site-packages\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\Python27\lib\site-packages\requests\sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "c:\Python27\lib\site-packages\requests\adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(10053, 'An established connection was aborted by the software in your host machine'))

Solution

  • Windows is closing the connection for an unspecified reason. See this other answer for more detail.

    In short, this is neither the fault of requests, requests-toolbelt, or pyramid.