I use libmnl to receive packets directed to user-space with the iptables "-j NFQUEUE". It works well. Here is the main loop;
for (;;) {
ret = mnl_socket_recvfrom(nl, buf, sizeof_buf);
if (ret == -1) {
perror("mnl_socket_recvfrom");
exit(EXIT_FAILURE);
}
ret = mnl_cb_run(buf, ret, 0, portid, queue_cb, nl);
if (ret < 0){
perror("mnl_cb_run");
exit(EXIT_FAILURE);
}
}
It has been observed that mnl_socket_recvfrom() does not return immediately when a packet is received. Up to 100ms delay has happened. To troubleshoot I use ping and tried to hog the CPU to provoke delays. But to my surprise the latency went down when the cpu hog was started!
64 bytes from 10.0.0.0: icmp_seq=120 ttl=63 time=1.08 ms
64 bytes from 10.0.0.0: icmp_seq=121 ttl=63 time=1.26 ms
64 bytes from 10.0.0.0: icmp_seq=122 ttl=63 time=1.15 ms
64 bytes from 10.0.0.0: icmp_seq=123 ttl=63 time=1.26 ms
64 bytes from 10.0.0.0: icmp_seq=124 ttl=63 time=1.24 ms
64 bytes from 10.0.0.0: icmp_seq=125 ttl=63 time=0.627 ms <--- CPU hog started
64 bytes from 10.0.0.0: icmp_seq=126 ttl=63 time=0.334 ms
64 bytes from 10.0.0.0: icmp_seq=127 ttl=63 time=0.628 ms
64 bytes from 10.0.0.0: icmp_seq=128 ttl=63 time=0.599 ms
64 bytes from 10.0.0.0: icmp_seq=129 ttl=63 time=0.602 ms
Can any body explain this?
The test is made on a kvm VM with one (virtual) CPU. The nfqueue program runs with normal prio and sends verdict accept and a fwmark to forward all packets (icmp in this case). It does not matter if the CPU-hog is running with normal or nice -20 prio.
The hog is written i C, and completely dumb:
unsigned long i = 0;
for (;;) {
i++;
}
This is because the CPUfreq governor "ondemand" sets the CPU frequency depending on the current system load. When you run a ping command, the CPU frequency will increase. The solution may be to set the CPU frequency to the highest frequency via this command:
cpupower frequency-set --governor performance