I have a local running FastAPI app that upon receiving a request, starts a background task (using FastAPI background tasks) and upon completion writes the response to the database. This works well when the app is running locally when I start multiple background tasks around the same time with my requests and it successfully writes to my cloud/supabase database.
However, when I deploy this same app to my cloud run (Note I use 'cpu always allocated' instance with a 6cpu and 6 gb ram) 1 background task seem to be running okay, but when I start multiple tasks simultaneously with requests, it seems to stop processing after a while and nothing seems to be happening, it stops both background tasks, ultimately the cloud run container shuts down and database is also not populated. I looked at my container usage and my cpu and ram are sufficient enough for this, and even if not I expected it to start multiple instances to process.
Also, note I don't see any logs in my Cloud run console and have set adequate concurrency limit (40). I thought it might have to do with my supabase database connection, but considering it works locally with the same database it seems like I'm missing something on how cloud run seems to be working with FastAPI background tasks.
Note: It does not at any time exceeds cpu or memory, my timeouts are 3600 seconds which is quite enough. I tried adjusting the concurrency settings as well (set it to 1) but still this issue. I keep sending a polling request from the client for the status on the background task and that request I do see in my logs for quite sometime.
A Cloud Run container may shut down unexpectedly if it exceeds its resource limits (CPU, memory) or timeouts. To manage background processes, ensure that tasks are efficient, use appropriate concurrency settings, and consider increasing resource allocations or using asynchronous task queues like Celery.