I am trying to filter out the user-initiated HTTP GET requests from a PCAP file. That is the HTTP GET requests, initiated by the user by clicking a link or directly providing in the address bar. I know we can filter out few requests using the following header parameters.
Content Type : text/html
Connection : keep-alive
But how can we filter out the requests initated by the following
Neither iframe nor AJAX calls differ from normal requests, so you cannot detect them reliably. Some AJAX requests can be detected by a X-Requested-With: XMLHTTPRequest header, but because this is a header manually added by only some frameworks it is not enough do detect all AJAX requests.
And, I've seen enough AJAX request which match your rule for user initiated header (e.g. with a text/html response and keep-alive) and most iframe requests will match this rule too.
Also, there are enough user-initiated requests which do not match this rule, especially the ones without the keep-alive (there are load balancers which explicitly close connection after a single response, e.g. Connection: close) and others might result in text/xml+html or text/xml or other content types.