Search code examples
c++httplibcurlkeep-alive

Easy way to "nudge" a server to keep a connection open?

Okay, so a little context: I have an app running on an embedded system that sends a few different requests over HTTP (using libcurl in C++) at the following intervals: 5 minutes 15 minutes 1 hour 24 hours

My goal: Reduce data consumption (runs over cellular)

We have both client and server side TLS authentication, so the handshake is costly. The idea is that we use persistent connections (at least for the shorter interval files) to avoid doing the handshake every time.

Unfortunately, after much tinkering I've figured out that the server is closing the connection before the intervals pass. Maybe this is something we can extend? I'll have to talk to the server side guys.

I was under the impression that was the reason the "TCP keep-alive" packets existed, but supposedly those "check the connection" not "keep it open" like the name suggests.

My idea is this:

Have my app send a packet (as small as possible) every 2 minutes or so (however long the timeout is) to "nudge" the connection into staying open.

My questions are:

  1. Does that make any sense?
  2. I don't suppose there is an easy way to do this in libcurl is there?
  3. If so, how small could we get the request?
  4. Is there an even easier way to do it? My only issue here is that all the connection stuff "lives" in libcurl.

Thanks!

Solution

  • It would be easier to give a more precise answer if you gave a little more detail on your application architecture. For example, is it a RESTful API? Is the use of HTTP absolutely mandatory? If so, what HTTP server are you using (nginx, apache, ...)? Could you consider websockets as an alternative to plain HTTP?

    If you are at liberty to use something other than regular HTTP or HTTPs - and to use something other than libcurl on the client side - you would have more options.

    If, on the other hand, if you are constrained to both

    1. use HTTP (rather than a raw TCP connection or websockets), and
    2. use libcurl

    then I think your task is a good bit more difficult - but maybe still possible.

    One of your first challenges is that the typical timeouts for a HTTP connection are quite low (as low as a few seconds for Apache 2). If you can configure the server you can increase this.

    I was under the impression that was the reason the "TCP keep-alive" packets existed, but supposedly those "check the connection" not "keep it open" like the name suggests.

    Your terminology is ambiguous here. Are you referring to TCP keep-alive packets or persistent HTTP connections? These don't necessarily have anything to do with each other. The former is an optional mechanism in TCP (which is disabled by default). The latter is an application-layer concept which is specific to HTTP - and may be used regardless of whether keep-alive packets are being used at the transport layer.

    My only issue here is that all the connection stuff "lives" in libcurl.

    The problem with using libcurl is that it first and foremost a transfer library. I don't think it is tailored for long-running, persistent TCP connections. Nonetheless, according to Daniel Stenberg (the author of libcurl), the library will automatically try to reuse existing connections where possible - as long as you re-use the same easy handle.

    If so, how small could we get the request?

    Assuming you are using a 'ping' endpoint on your server - which accepts no data and returns a 204 (success but no content) response, then the overhead - in the application layer - would be the size of the HTTP request headers + the size of the HTTP response headers. Maybe you could get it down to 200-300 bytes, or thereabouts.

    Alternatives to (plain) HTTP

    If you are using a RESTful API, this paradigm sort of goes against the idea of a persistent TCP connection - although I can not think of any reason why it would not work.

    You might consider websockets as an alternative, but - again - libcurl is not ideal for this. Although I know very little about websockets, I believe they would offer some advantages.

    Compared to plain HTTP, websockets offer:

    • significantly less overhead than HTTP per message;
    • the connection is automatically persistent: there is no need to send extra 'keep alive' messages to keep it open;

    Compared to a raw TCP connection, the benefits of websockets are that:

    • you don't have to open a custom port on your server;
    • it automatically handles the TLS/SSL stuff for you.

    (Someone who knows more about websockets is welcome to correct me on some of the above points - particularly regarding TLS/SSL and keep alive messages.)

    Alternatives to libcurl

    An alternative to libcurl which might be useful here is the Mongoose networking library. It would provide you with a few different alternatives:

    1. use a plain TCP connection (and a custom application layer protocol),
    2. use a TCP connection and handle the HTTP requests yourself manually,
    3. use websockets - which it has very good support for (both as server and client).

    Mongoose allows you to enable SSL for all of these options also.