I want to write a program to log all the URLs visited on a computer, but standalone, so not as a Fiddler2 extension. Are there any libraries out there that do this already that I could include in my application (which I was intending to write in C# .Net, but I'm flexible as long as it's for Windows)? If not are there any that could at least facilitate reading information from HTTP packets? I want to analyse the urls on the fly. Thanks.
For that, you would have to sniff the traffic on the computer under analysis.
To achieve this, use pcap library. As you may want to use a higher level programming language as C# (or Java), there are a lot of wrappers available to facilitate the usage of pcap library. In Java (since I am more used to it), there is one wrapper called jNetPcap. It is open source and has a good documentation. See the example below to sniff the traffic of any of our NICs:
package org.jnetpcap.examples;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import org.jnetpcap.Pcap;
import org.jnetpcap.PcapIf;
import org.jnetpcap.packet.PcapPacket;
import org.jnetpcap.packet.PcapPacketHandler;
/**
* Here is the output generated by this example :
*
* Network devices found:
* #0: \Device\NPF_{BC81C4FC-242F-4F1C-9DAD-EA9523CC992D} [Intel(R) PRO/100 VE]
* #1: \Device\NPF_{E048DA7F-D007-4EEF-909D-4238F6344971} [VMware Virtual Ethernet Adapter]
* #2: \Device\NPF_{5B62B373-3EC1-460D-8C71-54AA0BF761C7} [VMware Virtual Ethernet Adapter]
* #3: \Device\NPF_GenericDialupAdapter [Adapter for generic dialup and VPN capture]
*
* Choosing 'Intel(R) PRO/100 VE) ' on your behalf:
* Received packet at Tue Nov 03 18:52:42 EST 2009 caplen=1362 len=1362 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=82 len=82 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=145 len=145 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=62 len=62 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=164 len=164 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=62 len=62 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=54 len=54 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=1073 len=1073 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=1514 len=1514 jNetPcap rocks!
* Received packet at Tue Nov 03 18:52:45 EST 2009 caplen=279 len=279 jNetPcap rocks!
*/
public class ClassicPcapExample {
/**
* Main startup method
*
* @param args
* ignored
*/
public static void main(String[] args) {
List<PcapIf> alldevs = new ArrayList<PcapIf>(); // Will be filled with NICs
StringBuilder errbuf = new StringBuilder(); // For any error msgs
/***************************************************************************
* First get a list of devices on this system
**************************************************************************/
int r = Pcap.findAllDevs(alldevs, errbuf);
if (r == Pcap.NOT_OK || alldevs.isEmpty()) {
System.err.printf("Can't read list of devices, error is %s", errbuf
.toString());
return;
}
System.out.println("Network devices found:");
int i = 0;
for (PcapIf device : alldevs) {
String description =
(device.getDescription() != null) ? device.getDescription()
: "No description available";
System.out.printf("#%d: %s [%s]\n", i++, device.getName(), description);
}
PcapIf device = alldevs.get(0); // We know we have atleast 1 device
System.out
.printf("\nChoosing '%s' on your behalf:\n",
(device.getDescription() != null) ? device.getDescription()
: device.getName());
/***************************************************************************
* Second we open up the selected device
**************************************************************************/
int snaplen = 64 * 1024; // Capture all packets, no trucation
int flags = Pcap.MODE_PROMISCUOUS; // capture all packets
int timeout = 10 * 1000; // 10 seconds in millis
Pcap pcap =
Pcap.openLive(device.getName(), snaplen, flags, timeout, errbuf);
if (pcap == null) {
System.err.printf("Error while opening device for capture: "
+ errbuf.toString());
return;
}
/***************************************************************************
* Third we create a packet handler which will receive packets from the
* libpcap loop.
**************************************************************************/
PcapPacketHandler<String> jpacketHandler = new PcapPacketHandler<String>() {
public void nextPacket(PcapPacket packet, String user) {
System.out.printf("Received packet at %s caplen=%-4d len=%-4d %s\n",
new Date(packet.getCaptureHeader().timestampInMillis()),
packet.getCaptureHeader().caplen(), // Length actually captured
packet.getCaptureHeader().wirelen(), // Original length
user // User supplied object
);
}
};
/***************************************************************************
* Fourth we enter the loop and tell it to capture 10 packets. The loop
* method does a mapping of pcap.datalink() DLT value to JProtocol ID, which
* is needed by JScanner. The scanner scans the packet buffer and decodes
* the headers. The mapping is done automatically, although a variation on
* the loop method exists that allows the programmer to sepecify exactly
* which protocol ID to use as the data link type for this pcap interface.
**************************************************************************/
pcap.loop(10, jpacketHandler, "jNetPcap rocks!");
/***************************************************************************
* Last thing to do is close the pcap handle
**************************************************************************/
pcap.close();
}
}
This example was extracted from the jNetPcap website. As you can see, you just have to customize the nextPacket() method to make what you are intending.
That would simply be:
public void nextPacket(PcapPacket packet, String user) {
Http http = new Http();
if (packet.hasHeader(http)) {
System.out.printf("Received packet at %s: %s\n",
new Date(packet.getCaptureHeader().timestampInMillis()),
http.Request.valueOf("Host")
);
}
Hope I've helped.