nmap avoid random results (increase precision)

I am using nmap to detect open 80 ports for a given network (on the web, not locally), the problem is that some hosts in that network are overloaded (they handle huge http traffic), so nmap is skipping some of these hosts, for example :

a first scan using nmap for host X gives : open tcp port detected (80);
second scan : no open port;
third scan : open tcp port detected (80);
...

And so on, it doesn't give persistent results, and I need to know precisely which hosts have open 80 ports (or other port),

What I tried is to increase timeout, increase/decrease aggressiveness ... I end up with the following nmap command :

nmap -sS -p 80 -T3 -v -n --min-rtt-timeout 60m --max-retries 30 --host-timeout 5m -Pn --open X.X.X.X

I tried different parameters combination, what I want is to make nmap retries multiple times for each host so it make sure it has closed 80 (ot other) port,

Any ideas?

Thanks in advance,

Solution

I doubt that those servers are 'overloaded'. I would rather assume that the servers have some IDS running, could be a fail2ban or CSD/LFD setup, or maybe a firewall box or some appliance, that is thwarting the port scanning and thus frustrating your reconnaissance efforts. Any server that is exposed on the Internet must have some kind of protection, so you can expect that a lot of servers are really configured to make your job difficult.

Port scanning, especially doing it stealthily and reliably bypassing IDS is an art. Doing it in a crude fashion will return crude and somewhat unreliable results.

It is quite easy for an IDS to recognize the signature of an nmap scan like yours, that doesn't use any particular options - and block it, but there are techniques that exist to make it less obvious (see for example the nmap manual for some tips).

Some suggestions:

try to spread the scan across many different IP ranges/addresses and in a random order - if you scan every host in a /16 network in a sequential manner, at a sustained pace, that is quite obvious, noisy and definitely not stealth to the firewall protecting that network
learn about the nmap command-line options
be patient

In short, you have to experiment a lot. Find some hosts that you are sure have port 80 open, and others that shouldn't, try different commands, repeat them until you get consistent results, and don't forget timing is important, that means be gentle.

It depends on how many hosts you want to scan, or how large is the IP range you want to explore. If you are scanning the whole IPv4 space, the scalability issues are not the same as if you were scanning a home network for instance.

There is no one-size-fits-all solution, each server can have a different network setup and security settings, therefore they will not all react the same to an outside probe.

There are many tools available, and nmap is surely in the top 10 list that every penetration tester should be proficient with. But it is not the only one. Getting the most adequate tool for a specific job makes the task easier (or less difficult).

I have the feeling that Scapy could be a good candidate for this particular job. Since you are scanning port 80, at least try to make it look like a legitimate request from a browser. One thing you can try is use Wireshark on your own computer and sniff some HTTP requests to see what the packets from your web browser look like. Then try to 'replay' similar packets using Scapy or perhaps some other tool (hping ?). Then you can fire out packets that look like the 'real thing'.

Again, nmap is absolutely great, but you don't have a lot of control on the packets themselves.