Search code examples
c++multithreadingzeromq

ZeroMQ background thread creation


I am trying to understand the threading model of ZeroMQ.

According to their white paper http://zeromq.org/whitepapers:architecture#toc3, each I/O thread that the context is created with maps directly to an OS thread. What I don't understand is why two background threads are spawned by this minimal program:

#include <zmq.hpp>

int main() {
    zmq::context_t context{1};
    zmq::socket_t socket(context, ZMQ_SUB);
    socket.setsockopt(ZMQ_SUBSCRIBE, "", 0);
    socket.connect("tcp://localhost:5555");

}

As evidenced by the following gdb output, the line that creates the socket spawns two background threads.

Breakpoint 3, 0x00007f3cbebfbc80 in clone () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f3cbebfbc80 in clone () from /lib64/libc.so.6
#1  0x00007f3cbdff7f7a in do_clone.constprop () from /lib64/libpthread.so.0
#2  0x00007f3cbdff9469 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f3cbf74547d in zmq::thread_t::start(void (*)(void*), void*) () from /lib64/libzmq.so.5
#4  0x00007f3cbf70044a in zmq::ctx_t::start_thread(zmq::thread_t&, void (*)(void*), void*) const () from /lib64/libzmq.so.5
#5  0x00007f3cbf7012c1 in zmq::ctx_t::create_socket(int) () from /lib64/libzmq.so.5
#6  0x0000000000401251 in zmq::socket_t::init (this=0x7ffd5d0323f0, context_=..., type_=2) at /usr/include/zmq.hpp:649
#7  0x00000000004010a1 in zmq::socket_t::socket_t (this=0x7ffd5d0323f0, context_=..., type_=2) at /usr/include/zmq.hpp:463
#8  0x0000000000400e2f in main () at sub.cpp:5
(gdb) c
Continuing.
[New Thread 0x7f3cbc7e6700 (LWP 15499)]

Breakpoint 3, 0x00007f3cbebfbc80 in clone () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f3cbebfbc80 in clone () from /lib64/libc.so.6
#1  0x00007f3cbdff7f7a in do_clone.constprop () from /lib64/libpthread.so.0
#2  0x00007f3cbdff9469 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f3cbf74547d in zmq::thread_t::start(void (*)(void*), void*) () from /lib64/libzmq.so.5
#4  0x00007f3cbf70044a in zmq::ctx_t::start_thread(zmq::thread_t&, void (*)(void*), void*) const () from /lib64/libzmq.so.5
#5  0x00007f3cbf70135f in zmq::ctx_t::create_socket(int) () from /lib64/libzmq.so.5
#6  0x0000000000401251 in zmq::socket_t::init (this=0x7ffd5d0323f0, context_=..., type_=2) at /usr/include/zmq.hpp:649
#7  0x00000000004010a1 in zmq::socket_t::socket_t (this=0x7ffd5d0323f0, context_=..., type_=2) at /usr/include/zmq.hpp:463
#8  0x0000000000400e2f in main () at sub.cpp:5
(gdb) c
Continuing.
[New Thread 0x7f3cbbfe5700 (LWP 15500)]

Breakpoint 4, main () at sub.cpp:6
6       socket.setsockopt(ZMQ_SUBSCRIBE, "", 0);
(gdb) info thread
  Id   Target Id         Frame 
  3    Thread 0x7f3cbbfe5700 (LWP 15500) "a" 0x00007f3cbebfc2c3 in epoll_wait () from /lib64/libc.so.6
  2    Thread 0x7f3cbc7e6700 (LWP 15499) "a" 0x00007f3cbebfc2c3 in epoll_wait () from /lib64/libc.so.6
* 1    Thread 0x7f3cbfb4e840 (LWP 15498) "a" main () at sub.cpp:6
(gdb)

Using ZeroMQ and cppzmq-devel 4.1.4
Make with: g++ -std=c++11 -g sub.cpp -lzmq


Solution

  • why two background threads are spawned by this minimal program ?

    ZeroMQ is a broker-less asynchronous messaging infrastructure, built around the Context()-instance(s). If not familiar with the ZeroMQ way, one may enjoy this 5-second read about the main conceptual elements in ZeroMQ hierarchy in less than a five seconds section.

    Given the fact, there is always as chance to define how many IO-threads are to be spawned in a Context( nIOthreads = 1 ) to be instantiated, lets put it short:

    • a zero-IO-thread equipped Context( 0 )
      will spawn itself as a thread + 0 IO-threads, i.e. 1 thread
    • a solo-IO-thread equipped Context( 1 )
      will spawn itself as a thread + 1 IO-thread, i.e. 2 threads
    • a more-IO-threads equipped Context( N )
      will spawn itself as a thread + N IO-thread, i.e. 1+N threads

    given the experiment above, the real composition in-vivo could get easily validated.

    One can easily imagine the role of threads' specialisation inside the Context() instance, where transport-class related issues get separated from Context()'s main taks, i.e. doing the house-keeping + managing all the FSA(s) / workflow(s) + holding the asynchronous communication services across its local-interface with the application-side code.

    In this very same manner, the transport-class inproc://, having no real transport-related device to control, can work inside an IO-thread-less Context() instance, where just memory-mapping works across the "borders" with all due local-process memory, yet without a need to operate any IO-thread.