I am trying to upload a large file (≥3GB) to my FastAPI server, without loading the entire file into memory, as my server has only 2GB of free memory.
Server side:
@app.post("/uploadfiles")
async def uploadfiles(upload_file: UploadFile = File(...):
pass
Client side:
file_name="afd.tgz"
m = MultipartEncoder(fields = {"upload_file":open(file_name,'rb')})
prefix = "http://xxx:5000"
url = "{}/v1/uploadfiles".format(prefix)
try:
req = requests.post(
url,
data=m,
verify=False,
)
which returns the following 422 (Unprocessable entity) error:
HTTP 422 {"detail":[{"loc":["body","upload_file"],"msg":"field required","type":"value_error.missing"}]}
I am not sure what MultipartEncoder actually sends to the server, so that the request does not match. Any ideas?
With requests-toolbelt, you have to pass the filename as well, when declaring the field for upload_file, as well as set the Content-Type header—which is the main reason for the error you get, as you are sending the request without setting the Content-Type header to multipart/form-data, followed by the necessary boundary string—as shown in the docs. Example:
filename = 'my_file.txt'
m = MultipartEncoder(fields={'upload_file': (filename, open(filename, 'rb'))})
r = requests.post(url, data=m, headers={'Content-Type': m.content_type})
print(r.request.headers) # confirm that the 'Content-Type' header has been set
However, I wouldn't recommend using requests-toolbelt, as it hasn't provided a new release for over three years now. I would suggest using Python requests instead, as demonstrated in this answer and this answer (also see Streaming Uploads and Chunk-Encoded Requests), or, preferably, use httpx, which supports sending requests asynchronously (if you had to send multiple requests simultaneously), as well as streaming File uploads by default, meaning that only one chunk at a time would be loaded into memory (see the docs).
.stream()The example below demonstrates an approach, which was initially presented in this answer, on how to upload a file in a fast way compared to the one documented by FastAPI. As previously explained in the linked answer above, when you declare an UploadFile object, FastAPI/Starlette, under the hood, uses a SpooledTemporaryFile with the max_size attribute set to 1MB, meaning that the file data is spooled in memory, until the file size exceeds the max_size, at which point the contents will be written to disk; more specifically, the file data will be written to a temporary file on your OS's temporary directory—see this answer on how to find/change it—that you later need to read the data from, using the .read() method. Hence, this whole process makes uploading a file quite slow; especially, if it is a large one (as you'll see in Option 3 later on).
To avoid that and speed up the process as well, as the linked answer above suggested, one could access the request body as a stream. As per Starlette documentation, if you use the request.stream() method, the (request) byte chunks are provided without storing the entire body into memory (and later to a temporary file, if the body size exceeds 1MB). This method allows you to read and process the byte chunks as they arrive.
Even though the endpoint below is designed to only expect a single file, a client could make multiple calls to that endpoint (as demonstrated in the client examples below), in order to upload multiple files. Also, the endpoint below, compared to the ones from the other options later on, cannot accept Form data. One, however, could use the request headers—although it would be advisable to use the request body instead, as demonstrated in Options 2 and 3—in order to send some extra data (Note though that HTTP header values are restricted by server implementations; hence, be aware of the limits defined by the various web servers).
The example below saves the incoming files to disk, but if one would like having them stored to RAM instead, see the "Update" section of the linked answer above, where this approach was first presented. Also, the filenames are encoded/quoted on client side and decoded/unquoted on server side. This is to ensure that the uploading wouldn't fail, if one tried uploading files that their name had non-ascii/unicode characters in it.
app.py
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from urllib.parse import unquote
import aiofiles
import os
app = FastAPI()
templates = Jinja2Templates(directory="templates")
@app.post('/upload')
async def upload(request: Request):
try:
filename = request.headers['filename']
filename = unquote(filename)
filepath = os.path.join('./', os.path.basename(filename))
async with aiofiles.open(filepath, 'wb') as f:
async for chunk in request.stream():
await f.write(chunk)
except Exception:
raise HTTPException(status_code=500, detail='Something went wrong')
return {"message": f"Successfuly uploaded: {filename}"}
@app.get("/", response_class=HTMLResponse)
async def main(request: Request):
return templates.TemplateResponse(request=request, name="index.html")
templates/index.html
<!DOCTYPE html>
<html>
<body>
<label for="fileInput">Choose file(s) to upload</label>
<input type="file" id="fileInput" name="fileInput" onchange="reset()" multiple><br>
<input type="button" value="Submit" onclick="go()">
<p id="response"></p>
<script>
var resp = document.getElementById("response");
function reset() {
resp.innerHTML = "";
}
function go() {
var fileInput = document.getElementById('fileInput');
if (fileInput.files[0]) {
for (const file of fileInput.files) {
let reader = new FileReader();
reader.onload = function () {
uploadFile(reader.result, file.name);
}
reader.readAsArrayBuffer(file);
}
}
}
function uploadFile(contents, filename) {
var headers = new Headers();
filename = encodeURI(filename);
headers.append("filename", filename);
fetch('/upload', {
method: 'POST',
headers: headers,
body: contents,
})
.then(response => response.json()) // or, response.text(), etc.
.then(data => {
resp.innerHTML += JSON.stringify(data); // data is a JSON object
})
.catch(error => {
console.error(error);
});
}
</script>
</body>
</html>
test.py
import httpx
import time
from urllib.parse import quote
url = 'http://127.0.0.1:8000/upload'
filename = 'bigFile.zip'
headers = {'filename': quote(filename)}
start = time.time()
with open(filename, "rb") as f:
r = httpx.post(url=url, data=f, headers=headers)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.json())
To upload multiple files, you could use:
# ...
import glob, os
paths = glob.glob("big_files_dir/*", recursive=True)
for p in paths:
with open(p, "rb") as f:
headers = {'filename': quote(os.path.basename(p))}
# r = httpx...
File and Form data using .stream()The example below takes the suggested solution described above a step further, by using the streaming-form-data library, which provides a Python parser for parsing streaming multipart/form-data input chunks. This means that one not only can upload Form data along with File(s), but also the backend wouldn't have to wait for the entire request body to be received, in order to start parsing the data (as is the case with Option 3 below)—in other words, the parser would parse the data as they arrive, and that's what makes it fast.
The way it is done is that you initialize the main parser class (passing the HTTP request headers that help determine the input Content-Type, and hence, the boundary used to separate each body part in the multipart payload, etc.), and associate one of the Target classes to define what should be done with a field, when it has been extracted out of the request body. For instance, FileTarget would stream the data to a file on disk, whereas ValueTarget would hold the data in memory (the ValueTarget class can be used for either Form or File data, if you don't need the file(s) saved to the disk). It is also possible to define your own custom Target classes. It should be noted that streaming-form-data does not currently support async calls to I/O operations, meaning that the writing of chunks is synchronous (within a def function). Though, as the endpoint in the example below uses .stream() (which is an async def function), it will give up time for other tasks/requests in the event loop to run, while waiting for data to become available from the stream. You could also run the function for parsing the received data in a separate thread and await it, using Starlette's run_in_threadpool()—e.g., await run_in_threadpool(parser.data_received, chunk)—which is internally used by FastAPI, when you make calls to the async methods of UploadFile, as shown here. For more details on def vs async def in FastAPI, see this answer.
Using the solution below would also allow one to perform certain validation tasks, e.g., ensuring that the data size is not exceeding a certain limit—which couldn't be done with the UploadFile approach while data are streaming, as with UploadFile, the request gets inside the endpoint, after the file is fully uploaded. The solution below achieves that using MaxSizeValidator. However, as this would only be applied to File/Form fields that you had defined—and hence, it wouldn't prevent a malicious user from sending an extremely large request body (using random File/Form fields, for instance), which could result in consuming server resources in a way that the application may end up crashing or become unresponsive to legitimate users—the example below incorporates a custom MaxBodySizeValidator that could be used to ensure that the request body size does not exceed a pre-defined maximum value. Both validators desribed above solve the issue of limiting upload file size, as well as the entire request body size, in a likely better way than the one desribed here, which instead uses the UploadFile approach that requires the file to be entirely received and saved to the temporary directory, before performing any validation checks (not to mention that the approach described in that github post does not take into account the request body size at all, which makes the approach vulnerable to the attack mentioned earlier, where malicious actors may attempt to overload the server with excessively large requests). Using an ASGI middleware like this could be an alternative solution. Also, in case you are using Gunicorn with Uvicorn, you could also define limits, regarding the number of HTTP header fields in a request, the size of an HTTP request header field, etc. (see the docs). Similar limits could also be applied when using reverse proxy servers, such as Nginx (which also allows you to set the maximum request body size using the client_max_body_size directive).
A few notes for the example below. Since this approach uses the Request object directly, and not UploadFile/Form, the endpoint won't be properly documented in the Swagger auto-generated docs at /docs (if that's important for your application at all). This also means that you have to perform some validation checks on your own, such as whether the required fields for the endpoint were received or not, and if yes, whether they were in the expected format. For instance, for the data field, you could check whether the data.value is empty or not (empty would mean that the user has either not included that field in the multipart/form-data, or sent an empty value), as well as if isinstance(data.value, str). As for the file(s), you could check whether file_.multipart_filename is not empty; however, since a filename could likely not be included in the Content-Disposition by the user in their client request, you would also may want to check if the file exists in the filesystem, using os.path.isfile(filepath), in order to ensure that a file has been indeed uploaded (Note: you need to make sure there is no pre-existing file with the same name in that specified location; otherwise, the aforementioned function would always return True, even when the user did not send the file. You could always generate unique UUIDs for the filenames, as suggested here and here).
Regarding the applied size limits, the MAX_REQUEST_BODY_SIZE below must be larger than MAX_FILE_SIZE (plus all of the Form values size) you expcect to receive, as the raw request body (that you get from using the .stream() method) includes a few more bytes for the --boundary and Content-Disposition header for each of the fields in the body. Hence, you should add a few more bytes, depending on the Form values and the number of files you expect to receive (hence the MAX_FILE_SIZE + 1024 below).
app.py
from fastapi import FastAPI, Request, HTTPException, status
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, ValueTarget
from streaming_form_data.validators import MaxSizeValidator
import streaming_form_data
from starlette.requests import ClientDisconnect
from urllib.parse import unquote
import os
MAX_FILE_SIZE = 1024 * 1024 * 1024 * 4 # = 4GB
MAX_REQUEST_BODY_SIZE = MAX_FILE_SIZE + 1024
app = FastAPI()
class MaxBodySizeException(Exception):
def __init__(self, body_len: str):
self.body_len = body_len
class MaxBodySizeValidator:
def __init__(self, max_size: int):
self.body_len = 0
self.max_size = max_size
def __call__(self, chunk: bytes):
self.body_len += len(chunk)
if self.body_len > self.max_size:
raise MaxBodySizeException(body_len=self.body_len)
@app.post('/upload')
async def upload(request: Request):
body_validator = MaxBodySizeValidator(MAX_REQUEST_BODY_SIZE)
filename = request.headers.get('filename')
if not filename:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail='Filename header is missing')
try:
filename = unquote(filename)
filepath = os.path.join('./', os.path.basename(filename))
file_ = FileTarget(filepath, validator=MaxSizeValidator(MAX_FILE_SIZE))
data = ValueTarget()
parser = StreamingFormDataParser(headers=request.headers)
parser.register('file', file_)
parser.register('data', data)
async for chunk in request.stream():
body_validator(chunk)
parser.data_received(chunk)
except ClientDisconnect:
print("Client Disconnected")
except MaxBodySizeException as e:
raise HTTPException(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
detail=f'Maximum request body size limit ({MAX_REQUEST_BODY_SIZE} bytes) exceeded ({e.body_len} bytes read)')
except streaming_form_data.validators.ValidationError:
raise HTTPException(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
detail=f'Maximum file size limit ({MAX_FILE_SIZE} bytes) exceeded')
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='There was an error uploading the file')
if not file_.multipart_filename:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail='File is missing')
print(data.value.decode())
print(file_.multipart_filename)
return {"message": f"Successfuly uploaded {filename}"}
As mentioned earlier, to upload the data (on client side), you can use the HTTPX library, which supports streaming file uploads by default, and thus allows you to send large streams/files without loading them entirely into memory. You can pass additional Form data as well, using the data argument. Below, a custom header, i.e., filename, is used to pass the filename to the server, so that the server instantiates the FileTarget class with that name (you could use the X- prefix for custom headers, if you wish; however, it is not officially recommended anymore).
test.py
import httpx
import time
from urllib.parse import quote
url ='http://127.0.0.1:8000/upload'
filename = 'bigFile.zip'
files = {'file': open(filename, 'rb')}
headers = {'filename': quote(filename)}
data = {'data': 'Hello World!'}
with httpx.Client() as client:
start = time.time()
r = client.post(url, data=data, files=files, headers=headers)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.status_code, r.json(), sep=' ')
Files and Form data using .stream()One way to upload multiple files would be to perform multiple HTTP requests to that endpoint, one for each file, as explained in Option 1 earlier.
Another way, however, since the streaming-form-data package allows defining multiple files and form data, would be to use a header for each filename, or use random names on server side—the filename is needed, as the parser requires pre-defining the filepath for the FileTarget() class—and initialize the FileTarget class for each file (as explained earlier, if you don't need the file to be saved to disk, you could use ValueTarget instead). If you chose using random names, once the file is fully uploaded (when no more chunks left in request.stream()), you could optionally rename it to file_.multipart_filename (if available), using os.rename(). Regardless, in a real-world scenario, you should never trust the filename (or even the file extension) passed by the user, as it might be malicious, trying to extract or replace files in your system, and thus, it is always a good practice to add some random alphanumeric characters at the end/front of the filename, if not using a completely random name, for each file that is uploaded.
On client side, in Python, you should pass a list of files, as described in the httpx's documentation. Note that you should use a different key/name for each file, so that they don't overlap when parsing them on server side, e.g., files = [('file0', open('bigFile.zip', 'rb')),('file1', open('otherBigFile.zip', 'rb'))].
You could also test the example below, using the HTML template at /, which uses JavaScript to prepare and send the request with multiple files.
For simplicity purposes, the example below does not perform any validation checks on the body size; however, if you wish, you could still perform those checks, using the code provided in the previous example.
app.py
from fastapi import FastAPI, Request, HTTPException, status
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from starlette.requests import ClientDisconnect
from urllib.parse import unquote
import streaming_form_data
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, ValueTarget
import os
app = FastAPI()
templates = Jinja2Templates(directory="templates")
@app.get("/", response_class=HTMLResponse)
async def main(request: Request):
return templates.TemplateResponse(request=request, name="index.html")
@app.post('/upload')
async def upload(request: Request):
try:
parser = StreamingFormDataParser(headers=request.headers)
data = ValueTarget()
parser.register('data', data)
headers = dict(request.headers)
filenames = []
i = 0
while True:
filename = headers.get(f'filename{i}', None)
if filename is None:
break
filename = unquote(filename)
filenames.append(filename)
filepath = os.path.join('./', os.path.basename(filename))
file_ = FileTarget(filepath)
parser.register(f'file{i}', file_)
i += 1
async for chunk in request.stream():
parser.data_received(chunk)
except ClientDisconnect:
print("Client Disconnected")
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='There was an error uploading the file')
print(data.value.decode())
return {"message": f"Successfuly uploaded {filenames}"}
templates/index.html
<!DOCTYPE html>
<html>
<body>
<input type="file" id="fileInput" name="files" onchange="reset()" multiple><br>
<input type="button" value="Submit" onclick="submitUsingFetch()">
<p id="response"></p>
<script>
var resp = document.getElementById("response");
function reset() {
resp.innerHTML = "";
}
function submitUsingFetch() {
var fileInput = document.getElementById('fileInput');
if (fileInput.files[0]) {
var formData = new FormData();
var headers = new Headers();
formData.append("data", "Hello World!");
var i = 0;
for (const file of fileInput.files) {
filename = encodeURI(file.name);
headers.append(`filename${i}`, filename);
formData.append(`file${i}`, file, filename);
i++;
}
fetch('/upload', {
method: 'POST',
headers: headers,
body: formData,
})
.then(response => response.json()) // or, response.text(), etc.
.then(data => {
resp.innerHTML = JSON.stringify(data); // data is a JSON object
})
.catch(error => {
console.error(error);
});
}
}
</script>
</body>
</html>
test.py
To automatically load multiple files, see the clients in this answer and this answer.
import httpx
import time
from urllib.parse import quote
url ='http://127.0.0.1:8000/upload'
filename0 = 'bigFile.zip'
filename1 = 'otherBigFile.zip'
headers = {'filename0': quote(filename0), 'filename1': quote(filename1)}
files = [('file0', open(filename0, 'rb')), ('file1', open(filename1, 'rb'))]
data = {'data': 'Hello World!'}
with httpx.Client() as client:
start = time.time()
r = client.post(url, data=data, files=files, headers=headers)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.status_code, r.json(), sep=' ')
Files and JSON bodyIn case you would like to upload both file(s) and JSON instead of Form data, you could use the approach described in Method 3 of this answer, thus also saving you from performing manual checks on the received Form fields, as explained earlier (see the linked answer for more details). To that end, please make the following changes in the code above. For an HTML/JS example, please refer to this answer.
app.py
#...
from fastapi import Form
from pydantic import BaseModel, ValidationError
from typing import Optional
from fastapi.encoders import jsonable_encoder
#...
class Base(BaseModel):
name: str
point: Optional[float] = None
is_accepted: Optional[bool] = False
def checker(data: str = Form(...)):
try:
return Base.model_validate_json(data)
except ValidationError as e:
raise HTTPException(detail=jsonable_encoder(e.errors()), status_code=status.HTTP_422_UNPROCESSABLE_ENTITY)
@app.post('/upload')
async def upload(request: Request):
#...
# place the below after the try-except block in the example given earlier
model = checker(data.value.decode())
print(dict(model))
test.py
#...
import json
data = {'data': json.dumps({"name": "foo", "point": 0.13, "is_accepted": False})}
#...
File and Form data using FastAPI's UploadFile and FormThis option, for the reasons outlined in the beginning of this answer, is much slower than the previous two. This approach is similar to using await request.form(), as demonstrated in this answer and Option 1 of this answer.
If you would like to use a normal def endpoint instead, see this answer.
app.py
from fastapi import FastAPI, File, UploadFile, Form, HTTPException, status
import aiofiles
import os
CHUNK_SIZE = 1024 * 1024 # adjust the chunk size as desired
app = FastAPI()
@app.post("/upload")
async def upload(file: UploadFile = File(...), data: str = Form(...)):
try:
filepath = os.path.join('./', os.path.basename(file.filename))
async with aiofiles.open(filepath, 'wb') as f:
while chunk := await file.read(CHUNK_SIZE):
await f.write(chunk)
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='There was an error uploading the file')
finally:
await file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
As mentioned earlier, using this option would take longer for the file upload to complete, and as HTTPX uses a default timeout of 5 seconds, you will most likely get a ReadTimeout exception when the file is rather large, as the server will need some time to read the SpooledTemporaryFile in chunks and write the contents to a permanent location on the disk. Thus, you can configure the timeout (see the Timeout class in the source code too), and more specifically, the read timeout, which "specifies the maximum duration to wait for a chunk of data to be received (for example, a chunk of the response body)". If set to None instead of some positive numerical value, there will be no timeout on read.
test.py
import httpx
import time
url ='http://127.0.0.1:8000/upload'
files = {'file': open('bigFile.zip', 'rb')}
data = {'data': 'Hello World!'}
timeout = httpx.Timeout(None, read=180.0)
with httpx.Client(timeout=timeout) as client:
start = time.time()
r = client.post(url, data=data, files=files)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.status_code, r.json(), sep=' ')