Search code examples
dockerflaskmesosdcos

Flask request.data is slow

We have a few clients in Asia and the USA and we're seeing this strange behavior when calling request.data when handling their POST requests:

  • The Singapore client is super fast (> 10 ms)
  • The USA clients are not as fast (50 - 100 ms)
  • The Chinese client is the slowest (200+ ms)

We got the above data by using cProfile, so that should be accurate (I think?). The payload of each client varies between 50 - 700 bytes but does not seem to exhibit any patterns (the Singapore client has a medium sized POST payload and the Chinese one has a small sized one)

After looking in this question, I suspect we're facing something similar, where the request is processed immediately after the headers are received, so calling request.data blocks until the full POST payload is received. I am guessing that the Chinese clients are the slowest since the GFW slows down the transmission of the POST payload.

I have two questions:

  1. Does the analysis make sense?
  2. How can I fix this? The above behavior seems quite inefficient since my API instance is blocked for an additional amount of time and wastes CPU cycles. It seems like it would work better if the request was fully received before being sent to the API instance

FWIW, I inherited this code base and there may be some gaps in my understanding but our DCOS architecture is similar to the image below. I tried looking for configuration options in the external marathon LB to increase buffering or send only fully received requests but I didn't find such options.

DCOS architecture

Solution

  • Looks like I figured this one out!

    Apparently Marathon LB is a wrapper around HAProxy and HAProxy has a mechanism to receive the full HTTP request payload before forwarding it on to the backend. Adding the http-buffer-request option to the Marathon-LB configuration seems to have done the trick!