Search code examples
djangoamazon-s3broken-pipe

AWS broken pipe error when uploading files bigger than 1 MB


I am a django newbie, and I inherited a django back-end with little documentation. I am making a request to said server, which is hosted on AWS. To store the files in the request we use S3. I have found nothing on the django code that limits the size of the file uploads, and I suspect it may be AWS closing the connection because of file size.

This is the code I use, and below the error I get whenever the total size of the files is over 1 MB:

import requests
json_dict = {'key_1':'value_1','video':video,'image':,image}
requests.post('https://api.test.whatever.io/v1/register', json=dict_reg)

video is a video file ('.mov','.avi','.mp4',etc) with base64 encoding, and image is an image file ('.jpg','.png') with base64 encoding.

And this is the trace I get, ONLY when the total size is over 1 MB:

/usr/local/lib/python2.7/dist-
packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatfo
rmWarning: A true SSLContext object is not available. This prevents urllib3 
from configuring SSL
appropriately and may cause certain SSL connections to fail. You can upgrade 
to a newer version of Python to solve this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarningTraceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 110, in 
post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in 
request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 
488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 
609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 
473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 
'Broken pipe'))

As mentioned previously, I have not found anywhere in the django code a limit to the file size, any hints where I should be looking at? I also did not find anything on the AWS S3 policy.


Solution

  • Assuming you have a Nginx to reverse proxy your HTTP requests? if yes check this link.

    Also see the value set for the below value in settings for the Upload Handlers in django

    FILE_UPLOAD_MAX_MEMORY_SIZE