What is the drawback of cpp-httplib using blocking socket I/O

Question

We have been using the excellent cpp-httlib for making REST requests. We would now like to use it for a server. Right at the top of the documentation is:

Important

This library uses 'blocking' socket I/O. If you are looking for a library with 'non-blocking' socket I/O, this is not the one that you want.

I'm not sure what this means in a practical sense, as while I understand the basic principles of IP/UDP/TCP etc, I am unfamiliar with socket programming. I've tested the server by firing off multiple requests to it, e.g. if the first request is implemented by sleeping for 30s, a second "hello world" request still returns immediately.

Debugging the code, I see the main thread runs:

    while (svr_sock_ != INVALID_SOCKET) {
      // socket operations 

      if (!task_queue->enqueue(
              [this, sock]() { process_and_close_socket(sock); })) {
        detail::shutdown_socket(sock);
        detail::close_socket(sock);
      }
    }

Worker threads run

      for (;;) {
        std::function<void()> fn;
        {
          // Lock / wait for new request

          fn = pool_.jobs_.front();
          pool_.jobs_.pop_front();
        }
        fn();
      }

i.e. the main thread is like an event loop in a GUI application, taking each incoming request, but rather than executing the request itself, it puts it on a queue for worker threads to pick up.

Previously I was wrapping up calls to C++ in a Python FastAPI web server, which is fine, but running Python as well as C++ clearly adds overhead / complexity, and it's not entirely straightforward to get your head around how requests are executed w.r.t. global interpreter lock. The direct C++ approach using cpp-httlib looks a whole lot better, but the comment about 'blocking' socket I/O worries me. What am I missing out on by not using non-blocking socket I/O?

(The server will be built for both Windows and linux; however prod will be linux.)

Answer 1

A blocking IO library uses threads to do preemptive multitasking, while a non-blocking one does cooperative multitasking, also see What is the difference between cooperative multitasking and preemptive multitasking?

with pre-emptive multitasking (cpp-httplib) if you want 1000 sockets handled at once then you will need 1000 threads active, and most of the CPU time will be wasted on the OS scheduling instead of doing the actual work, a context switch could take a few microsconds, it could even take more time on a virtualized OS, which is used by cloud providers.

with cooperative multitasking (asio), if you have a server with 4 cores, you only need 4 threads to manage the 1000 sockets, the OS won't have to do any excessive scheduling, and your app will handle this extra scheduling but in user-land with less system calls, which eventually ends up faster. (because operating systems have complicated schedulers, while user-land schedulers are simple and therefore fast).

if you are doing heavy computations per task or managing a very limited number of sockets at once and you are not limited by IO then using pre-emptive multitasking (cpp-httplib) is okay, but if you are IO limited (such as an html or file server) then you should be using cooperative multi-tasking (asio).

cooperative multitasking (asio) could be harder to use, but it is faster, while pre-emptive multitasking (cpp-httplib) could be slower but it is easier to use.

for comparison FastApi uses co-operative multitasking for async paths and pre-emptive multitasking for non-async ones.