Using libpcap to gather statistics on connections

Question

I have a HTTP threading proxy, i.e. for every request from clients I spawn a thread: now I want to gather some stats like bits per second (bps) and packets per second (pps).

I like my code to do one thing only, so if a thread handles a connection it won't calculate bps and pps for every packet too, I'm leaving this to another thread.

I create a thread for every HTTP request from client and if the proxy successfully connects to the requested remote server, the proxt sends the actual HTTP request to server and before routing data, the connection thread creates a logging thread: logging thread will calculate bps and pps until the connection is open. Connection thread gives to logging thread infos about what packets to filter (local IP address, local port, remote IP address, remote port) so every logging thread will filter only packets from the parent connection thread.

I'm having problems calculating bps and pps for every packet.

Here's the pseudocode of my looping capture of packets in logging thread:

// pcap variables
pcap_t *handle;
struct pcap_pkthdr *header;
const u_char *pkt_data;
// timevals used to calculate delay from last filtered packet
struct timeval oldTimevalUpload;
struct timeval oldTimevalDownload;      
memset(&oldTimevalUpload, 0, sizeof(oldTimevalUpload));
memset(&oldTimevalDownload, 0, sizeof(oldTimevalDownload));

// stopLogging is a boolean flag declared in connection "parent" thread: 
// it is set to false when connection thread has done sending and
// receiving data and connection is going to be closed
while (((res = pcap_next_ex(handle, &header, &pkt_data)) >= 0) && (!stopLogging)) {
    // check res
    if (packet is upload) {
        struct timeval difference;
        timeval_subtract(&difference, &(header->ts), &oldTimevalUpload);
        long long delay = (difference.tv_sec * 1000000) + difference.tv_usec;
        long long acceptedPackets = ((long long)(pkt_data)) * 1000000;
        long long acceptedBits = ((long long)(pkt_data+8)) * 8 * 1000000;
        long long pps = acceptedPackets / delay;
        long long bps = acceptedBits / delay;
        debugRed(host << ", UPLOAD DIVIDE ACCEPTED PKTS " << acceptedPackets << 
            " AND ACCEPTED BITS " << acceptedBits << " PER DELAY " << delay << 
            " IS PPS " << pps << " AND BPS " << bps);
        oldTimevalUpload.tv_sec = header->ts.tv_sec;
        oldTimevalUpload.tv_usec = header->ts.tv_usec;
    } else if (packet is download) {
        // basically the same as above
    }
}
debug("Quit logging connection " << localIPaddr << ":" << localPort << " and "
    << remoteIPaddr << ":" << remotePort);
pcap_close(handle);

And here's a sample output:

www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1479811349890053 IS PPS 0 AND BPS 0
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1479811350032141 IS PPS 0 AND BPS 0
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 4344 IS PPS 22845174953 AND BPS 182761414364
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 146464 IS PPS 677568822 AND BPS 5420551015
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 2815 IS PPS 35253797513 AND BPS 282030402841
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 2808 IS PPS 35341680911 AND BPS 282733470085
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1120 IS PPS 88606642857 AND BPS 708853200000
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1134 IS PPS 87512733686 AND BPS 700101925925
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 39658 IS PPS 2502381360 AND BPS 20019052498
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 176317 IS PPS 562846690 AND BPS 4502773890
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 136687 IS PPS 726034224 AND BPS 5808274261

I never knew my home network could withstand over 500 MBps, so something must be wrong.

This page shows how to calculate bps and pps and explains the shift of 8 chars in acceptedBits, but I'm going to report it down anyway. Here you can see the second and third parameter of function pcap_next_ex:

Basically, I did all what he said! Why am I getting so large and strange bps and pps?

Working on Ubuntu 14.04; don't know how to check libpcap version, but locate libpcap gives this:

/home/dexter/Desktop/wireshark-1.99.9/wiretap/libpcap.c
/home/dexter/Desktop/wireshark-1.99.9/wiretap/libpcap.h
/usr/lib/x86_64-linux-gnu/libpcap.a
/usr/lib/x86_64-linux-gnu/libpcap.so
/usr/lib/x86_64-linux-gnu/libpcap.so.0.8
/usr/lib/x86_64-linux-gnu/libpcap.so.1.5.3
/usr/share/doc/libpcap-dev
/usr/share/doc/libpcap0.8
/usr/share/doc/libpcap0.8-dev
/usr/share/doc/libpcap-dev/changelog.Debian.gz
/usr/share/doc/libpcap-dev/copyright
/usr/share/doc/libpcap0.8/CREDITS.gz
/usr/share/doc/libpcap0.8/README.Debian
/usr/share/doc/libpcap0.8/README.gz
/usr/share/doc/libpcap0.8/changelog.Debian.gz
/usr/share/doc/libpcap0.8/copyright
/usr/share/doc/libpcap0.8-dev/changelog.Debian.gz
/usr/share/doc/libpcap0.8-dev/copyright
/var/lib/dpkg/info/libpcap-dev.[list,md5sums]
/var/lib/dpkg/info/libpcap0.8-dev.[list,md5sums,preinst] 
/var/lib/dpkg/info/libpcap0.8:amd64.[list,md5sums,postinst,postrm,shlibs,symbols]

Answer 1

In your code:

long long acceptedPackets = ((long long)(pkt_data)) * 1000000;
long long acceptedBits = ((long long)(pkt_data+8)) * 8 * 1000000;

While pkt_data is a pointer.

What you are doing is basically getting the address of the packet data, converting it to a long long, adding 8 (for the second line), multiplying that by a constant and considering that to be your value, which is semantically incorrect. You should dereference that pointer, taking into account your data type (convert the pkt_data to a pointer to long long).

In code:

long long acceptedPackets = (*(long long*)(pkt_data)) * 1000000;
long long acceptedBits = (*(long long*)(pkt_data+8)) * 8 * 1000000; 
// this also works:
//long long acceptedPackets = *(long long*)pkt_data * 1000000;
//long long acceptedBits = *((long long*)pkt_data + 1) * 8 * 1000000;

For an example, see http://ideone.com/JqmRre

EDIT:

From this guide:

The last argument is the most interesting of them all, and the most confusing to the average novice pcap programmer. It is another pointer to a u_char, and it points to the first byte of a chunk of data containing the entire packet

It means that pkt_data is the packet contents itself. Unless your first 16 bytes of the packet contains the desired information (which is untrue, since it contains the raw packet, so it has the ETH, IP and TCP/UDP headers) you cannot use that data. In order to get the PPS metrics, you would have to implement a simple counter in your loop (since your are printing that metric every frame, a simple long long pps = (long long)(1.0 / delay); should suffice - note that the division is in floating point. For your BPS metrics, you should use the frame header information. So long long bps = (long long)(header->caplen * 8.0 / delay); should do.

As a side note, for time metrics, since you are using C++11, try using chrono. It is clearer and safer than timeval:

Add a #include <chrono>.

Your final code should look something like this:

// pcap variables
pcap_t *handle;
struct pcap_pkthdr *header;
const u_char *pkt_data;
// Use of high_resolution_clock
std::chrono::high_resolution_clock::time_point oldTimeUpload = std::chrono::high_resolution_clock::now();
std::chrono::high_resolution_clock::time_point oldTimeDownload = std::chrono::high_resolution_clock::now();

// stopLogging is a boolean flag declared in connection "parent" thread: 
// it is set to false when connection thread has done sending and
// receiving data and connection is going to be closed
while (((res = pcap_next_ex(handle, &header, &pkt_data)) >= 0) && (!stopLogging)) {
    // check res
    if (packet is upload) {
        std::chrono::high_resolution_clock::time_point now = std::chrono::high_resolution_clock::now();
        long long delay = std::chrono::duration_cast<std::chrono::nanoseconds>(now - oldTimeUpload).count();
        long long pps = (long long)(1000000000.0 / delay);
        long long bps = (long long)(header->caplen * 8 * 1000000000.0 / delay);
        debugRed(host << ", UPLOAD DIVIDE PER DELAY " << delay << 
            " IS PPS " << pps << " AND BPS " << bps);
        oldTimevalUpload.tv_sec = header->ts.tv_sec;
        oldTimevalUpload.tv_usec = header->ts.tv_usec;
    } else if (packet is download) {
        // basically the same as above
    }
}
debug("Quit logging connection " << localIPaddr << ":" << localPort << " and "
    << remoteIPaddr << ":" << remotePort);
pcap_close(handle);