I know its sends out different TCP, UDP, and ICMP probes to known open and closed ports of the target machine. I am trying to use nmap to find out OS of all the computers in the network but they are behind the firewalls, so need to know which ports I need to open in the Firewall. Currently to fails to guess anything, and most of them are running some kind of windows version.
Nmap's OS fingerprinting is based on matching responses to unusual TCP, UDP, and ICMP probes. Nmap chooses which ports to use based on the results of its port scan phase, which is why you cannot detect a remote OS without performing some kind of port scan. In order to get a good match, you need to satisfy as many of these conditions as possible:
-sS or -sT to get this result)-sU to scan for UDP ports)These are ranked in order of importance. You cannot get an accurate match without a TCP scan, but you can occasionally come close even if the target is filtering all UDP ports or blocking ICMP Echo Requests.
It is important that all the replies to these probes come from the single system to be fingerprinted. Firewalls and other security devices often spoof replies, especially to some of the unusual packets that Nmap sends. Other devices will alter the packets, changing things like the Don't Fragment bit or the TTL, which can result in an inaccurate match.