I'm interested in making a bandwidth-control utility, and I need to limit outgoing or inbound traffic dynamically. To do this, I could use ipfw pipes, but this creates a dependency on ipfw. How do programs like ipfw control bandwidth, at the syscall / socket level? My goal is to implement similar functionality, tailored for my specific needs.
All bandwidth limiters work on a simple principle – by filtering (dropping) packets. The implementation however is much more complex and require good knowledge of the TCP/IP stack and the consequences of filtering different types of traffic.
Bandwidth limiters are modeled on something called a token bucket. Tokens (that represent the data to be sent or received) are added to the bucket on regular time intervals. Since the bucket has a limited capacity if the bucket is full, new tokens are discarded (they overflow). This bucket is used to manage the amount of traffic. If there is enough tokens in the bucket the number of tokens is reduced for appropriate amount (proportional to the size of the packet) and the packet is sent or received. If there are not enough tokens in the bucket the packet is discarded. This is a basic explanation, read more on wiki or some other resource.
There are two basic kinds of traffic amount management — policing and shaping. The basic difference between the two is that a shaper has a queue or a buffer that can hold some amount of data. If the packets can't be sent immediately due to low number of tokens packets are placed in this buffer and are sent after there is a enough tokens. This technique is commonly used for outgoing traffic while the policing is used for incoming traffic.
After that you should want to know more about different token bucket mechanisms, how to avoid global synchronization of TCP flows, what are the effects of RED/WRED on non-TCP traffic, and a lot of other stuff linked to queues and QoS. In other words I think that satisfying a dependency is more easier than implementing all this so I suggest you use the tools that already work and build what you need on top of them.