Does recv remove packets from pcaps buffer?

Question

Say there are two programs running on a computer (for the sake of simplification, the only user programs running on linux) one of which calls recv(), and one of which is using pcap to detect incoming packets. A packet arrives, and it is detected by both the program using pcap, and by the program using recv. But, is there any case (for instance recv() returning between calls to pcap_next()) in which one of these two will not get the packet?

I really don't understand how the buffering system works here, so the more detailed explanation the better - is there any conceivable case in which one of these programs would see a packet that the other does not? And if so, what is it and how can I prevent it?

Answer 1

AFAIK, there do exist cases where one would receive the data and the other wouldn't (both ways). It's possible that I've gotten some of the details wrong here, but I'm sure someone will correct me.

Pcap uses different mechanisms to sniff on interfaces, but here's how the general case works:

• A network card receives a packet (the driver is notified via an interrupt)
• The kernel places that packet into appropriate listening queues: e.g.,
  • The TCP stack.
  • A bridge driver, if the interface is bridged.
  • The interface that PCAP uses (a raw socket connection).
• Those buffers are flushed independently of each other:
  • As TCP streams are assembled and data delivered to processes.
  • As the bridge sends the packet to the appropriate connected interfaces.
  • As PCAP reads received packets.

I would guess that there is no hard way to guarantee that both programs receive both packets. That would require blocking on a buffer when it's full (and that could lead to starvation, deadlock, all kinds of problems). It may be possible with interconnects other than Ethernet, but the general philosophy there is best-effort.

Unless the system is under heavy-load however, I would say that the loss rates would be quite low and that most packets would be received by all. You can decrease the risk of loss by increasing the buffer sizes. A quick google search tuned this up, but I'm sure there's a million more ways to do it.

If you need hard guarantees, I think a more powerful model of the network is needed. I've heard great things about Netgraph for these kinds of tasks. You could also just install a physical box that inspects packets (the hardest guarantee you can get).