Search code examples
dockergoogle-cloud-platformtimeoutgoogle-cloud-runhttp-status-code-502

Cloud Run/Gunicorn giving 502 error after one minute


I'm deploying a python application in Google Cloud Run that uses Gunicorn. Both my gunicorn and cloud run timeout are set to 900 seconds, which is also the timeout for Cloud Run. Strangely, when I call the function, I get a 502 error from Cloud Run if the application runs for more than 60 seconds, and not if it runs less than 60 seconds. For example, the deployed function below threw this error:

def process_file(request=request):
    time.sleep(61)
    ...
    return handle_response()      

However, if I changed the sleep to 40 seconds:

def process_file(request=request):
    time.sleep(40)
    ...
    return handle_response() 

there was no 502 error. I thought at first that the issue was caused by nginx, which has a 60 second default timeout, but it does not seem like nginx is deployed with docker or cloud run by default, so this doesn't seem like the cause of the issue. My Dockerfile is below:

FROM continuumio/miniconda3

# Install production dependencies.
RUN conda install numpy==1.17.2
RUN conda install xlsxwriter==1.1.2
RUN conda install pandas==0.25.1
RUN conda install -c conda-forge ciso8601
RUN pip install gunicorn flask gevent flask_mail flask-cors pyjwt firebase_admin networkx datefinder google-cloud-pubsub 

# Copy local code to the container image.
COPY app.py .
RUN mkdir backend/
COPY backend/ /backend/

# Service must listen to $PORT environment variable.
# This default value facilitates local development.
ENV PORT 8080

# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
CMD exec gunicorn --bind 0.0.0.0:$PORT --workers 1 app:app --timeout 900 --log-level debug

I am calling the cloud run using axios in the frontend, which from my understanding does not have a timeout, so I do not believe that this should be an issue. Any help is appreciated, thanks!

EDIT: Here is an image of the error message in the chrome console - does not seem to be very helpful though:

enter image description here


Solution

  • Figured out the issue. I was sending HTTP POST requests to a Firebase hosted domain. Firebase hosted domain POST requests time out after 60 seconds (see Firebase-Hosted Cloud Function retrying on any request that takes 60s, even when timeout is >60s) - the solution was to call the Cloud Run url directly instead.