I've recently been reading about OpenOnload, which bypasses the Linux kernel during certain network operations to improve performance. This made me curious about how network data between LXC containers on the same server is handled. If I send data from one to another, does it go all the way through the kernel TCP stack multiple times? If so, how much latency does this add compared to something like shared memory (which I might use if I wasn't using LXC)? Is some sort of kernel bypass feasible for LXC-LXC communication?
As you know:
The Linux kernel comprises cgroups for resource isolation (CPU, memory, block I/O, network, etc.) that does not require starting any virtual machines. Cgroups also provides namespace isolation to completely isolate applications' view of the operating environment, including process trees, network, user ids and mounted file systems.
LXC combines cgroups and namespace support to provide an isolated environment for applications. You asked: This made me curious about how network data between LXC containers on the same server is handled.
It's like one process on linux mashine communicate to another/same process on the same mashine, but if he would use one or two linux bridge's (LXC network inside container) + those bridges inside different cgroups/namespace.
If I send data from one to another, does it go all the way through the kernel TCP stack multiple times? If so, how much latency does this add compared to something like shared memory (which I might use if I wasn't using LXC)? Is some sort of kernel bypass feasible for LXC-LXC communication?
It's not great latency but when I used php-fpm as daemon that listen at 127.0.0.1:8888 and changed it to listen the unix socket there was increase of rps(pages per second). Performance of unix sockets vs TCP ports
PS: You can use unix sockets on your LXC containers. For example I use to create shared dir between all containers ( for example: /tmp/mysql.sock ):
mount --bind /lxc/shared /lxc/shared
mount --make-unbindable /lxc/shared