Search code examples
pythonasync-awaitpython-asynciofastapi

how to run data processing tasks asynchronously in fast api?


I have a route in my fast api application, (sample code below). my root route calls an async function called read_root. in this function , i want to query database , gets some result and upload it to aws in a s3 bucket. I don't necessarily have to wait for the database operations to finish, as it may take time processing the data and uploading it to s3. I created another async function called call_database do the database call , processing and finally uploading the files to s3, but this operation is all synchronous, i.e calling database , getting results from database and processing data and then uploading to s3.

my question , is can function defined as async process such synchronous operations ? and while those are in process, is it ok to return the response ('hello': 'world') in my case , to the http call . this will end that http request/response. but i am assuming the call_database function will keep running until the files are uploaded to s3?

is this a correct understanding and implementation?

from typing import Union

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def read_root():
    await call_database()

    return {"Hello": "World"}


async def call_database():
   //syncrohonous call to database
   //get the result back , and upload it to aws s3

Solution

  • I think you're looking for background task

    from typing import Union
    
    from fastapi import FastAPI, BackgroundTasks
    
    app = FastAPI()
    
    def upload_to_s3(data: dict):
        # your upload process here
    
    @app.get("/")
    async def read_root(background_tasks: BackgroundTasks):
        data_to_upload = await call_database()
        background_tasks.add_task(upload_to_s3, data_to_upload)
        return {"Hello": "World"}
    
    
    async def call_database():
       //syncrohonous call to database