Search code examples
.nethttpdnshttprequestresolve

.NET - downloading multiple pages from a website with a single DNS query

I'm using HttpRequest to download several pages from a website (in a loop). Simplifying it looks like this: 

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(
            "http://sub.domain.com/something/" + someString
        );

HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

//do something

I'm not quite sure actually but every request seems to resolve the address again (I don't know how to test if I'm right). I would like to boost it a little and resolve the address once and then reuse it for all requests. I can't work out how to force HttpRequest into using it, though.

I have tried using Dns.GetHostAddresses, converting the result to a string and passing it as the address to HttpWebRequest.Create. Unfortunately, server returns error 404 then. I managed to google that's probably because the "Host" header of the http query doesn't match what the server expects.

Is there a simple way to solve this?

Solution

  • I doubt that the DNS isn't being cached already to be honest, but there is a way to do what you ask.

    After creating the request with the IP address, set the Host property on it to the DNS name. This should solve your 404 problem.

    Something that might help you speed up your multiple requests is to set the KeepAlive property to true. This will keep the connection open and allow you to make multiple requests without having to re-establish the connection each time.