Search code examples
socketscurltcpkeep-alive

Curl TCP keepalives on mac

I have multiple servers set up under a load balancer that distributes requests to them by TCP connection. In other words - if I issue many requests in the browser all of them will be sent to one of the servers that are under load balancer as the TCP connection is opened.

However when I issue requests via curl the TCP connections seem not to be reused and the load balancer send each request to a new server (round-robin algorithm).

QUESTIONS:

  1. Is it possible to enable TCP keepalives with CURL? If so - how?
  2. Should I use something from libcurl, like : http://curl.haxx.se/libcurl/c/persistant.html - how should I do it?
  3. Is it related to the fact that I use mac? http://sourceforge.net/p/curl/bugs/1214/

Thanks.

What I have tried:

for i in {1..100}; do curl --keepalive --keepalive-time 50 -s -D - http:URL -o /dev/null; done

while looping I run this and see that new port is used every time:

lsof -i -n -P | grep curl
Solution

  • This is not possible the way you envision. Since you are creating a new curl process for each URL this will result in a new TCP connection which will end with process close. So even if curl itself would use TCP keep-alive it would not matter because it would be active only until the process is done. Curl by itself will already try to re-use the same connection for multiple requests as long as these requests are inside the same process (like with redirects).

    What you would need instead is a way to process multiple URLs inside the same process, so that they could reuse the same TCP connection for multiple requests. This is not possible with the curl command line tool, since this can only process a single URL per run. You have to use instead a tool which could handle multiple URLs within the same process.

    Is it possible to enable TCP keepalives with CURL? If so - how?

    Yes it is possible but it will not help with your problem.

    Should I use something from libcurl, like : http://curl.haxx.se/libcurl/c/persistant.html - how should I do it?

    Yes, this could help because you could do multiple requests this way from within the same process. Bindings are available for different programming languages. You could also use instead the native and comfortable HTTP handling of various script languages like python, perl, ruby...

    Is it related to the fact that I use mac? http://sourceforge.net/p/curl/bugs/1214/

    No, since the problem itself can not be handled with TCP keep alive at all.