I am currently working with a Cloud Run API backend that processes incoming HTTP POST requests. When the system has been idle for some time, it goes into a "cold start". Upon receiving a new request during a cold start, the request is processed twice, resulting in duplicate actions. This duplication is not desirable and I am looking for a solution to prevent it.
Here is a simplified description of the system:
The API receives an HTTP POST request containing a message payload. The API processes the request, which involves making a search request to an external service and returning the search result. Despite this seemingly straightforward process, the logs show that during a cold start, the incoming request is processed twice.
Here's what I've tried so far:
Returning a response immediately: As soon as the POST request is received, I tried returning a response to the sender, and then proceeding to process the request in a background task. However, this did not solve the issue, as the request is still being processed twice.
Creating a temporary set and storing the message_ids from the post request so if I get the same message_id I can check with the set and ignore it.
So when I created the second mechanism I thought I solved it however it's not catching it.
Here is also a simplified version of my logs:
2023-06-15 21:00:58.675 BST
POST200660 B5 msUnknown https://my-cloud-run-app.com/message
2023-06-15 21:01:00.485 BST
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
Default STARTUP TCP probe succeeded after 1 attempt for container "message-1" on port 8080.
Incoming request: POST [{"type":"inbound","message_id":1935379776,...}]
2023-06-15 21:01:25.539 BST
Message Processed.
...
2023-06-15 21:02:16.550 BST
POST200660 B7 msUnknown https://my-cloud-run-app.com/message
2023-06-15 21:02:16.560 BST
Incoming request: POST [{"type":"inbound","message_id":1935379776,...}]
2023-06-15 21:02:16.563 BST
Message Processed.
...
2023-06-15 21:17:31.301 BST
INFO: Shutting down
2023-06-15 21:17:31.403 BST
INFO: Waiting for application shutdown.
2023-06-15 21:17:31.403 BST
INFO: Application shutdown complete.
2023-06-15 21:17:31.404 BST
INFO: Finished server process [1]
As you can see same post request comes twice and I'm checking the message_id but still It makes it.
One particularly interesting things which might help is I'm using 2 different 3rd party API. One is google_search. The other is for sending the message.
Although it shows that it process the Message twice It only sends one request to google_search but twice to SMS API
I solved the issue by adding a cpu-boost to my cold-start. This helped me to not to hit timeout so I didn't get duplicate messages.
You can add that to your gcloud deploy function as follows:
gcloud run deploy <YOUR SERVICE> --image etc... --cpu-boost