How To Make Parallel HTTP Requests

I have a console app that makes a single request to a web page and returns its server status eg. 200, 404, etc..

I'd like to apply following changes:

List of User Inputs:

Url to request
How many parallel connections to use(concurrent users)
How long(seconds) to submit as many requests as it can

List of Outputs:

Show Total Fetches
Show Fetches per Second
Show Average Response Time (ms)

I imagine the best way to do it is to run multiple http fetches in parallel and run in a single process, so it doesn't bog down the client machine.

I really like C# but I'm still new to it. I've researched other articles about this but I don't fully understand them so any help would be greatly appreciated.

My Code:

static void Main(string[] args)
{
        try
        {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://10.10.1.6/64k.html");
            webRequest.AllowAutoRedirect = false;
            HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
            //Returns "MovedPermanently", not 301 which is what I want.

            int i_goodResponse = (int)response.StatusCode;
            string s_goodResponse = response.StatusCode.ToString();
            Console.WriteLine("Normal Response: " + i_goodResponse + " " + s_goodResponse);
            Console.ReadLine();
        }
        catch (WebException we)
        {
            int i_badResponse = (int)((HttpWebResponse)we.Response).StatusCode;
            string s_badResponse = ((HttpWebResponse)we.Response).StatusCode.ToString();
            Console.WriteLine("Error Response: " + i_badResponse + " " + s_badResponse);
            Console.ReadLine();
        }
    }

Some possible code that I found:

void StartWebRequest()
{
    HttpWebRequest webRequest = ...;
    webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), webRequest);
}

void FinishWebRequest(IAsyncResult result)
{
    HttpWebResponse response = (result.AsyncState as HttpWebRequest).EndGetResponse(result) as HttpWebResponse;
}

Solution

This is actually a good place to make use of the Task Parallel Library in .NET 4.0. I have wrapped your code in a Parallel.For block which will execute a number of sets of requests in parallel, collate the total times in each parallel branch, and then calculate the overall result afterwards.

int n = 16;
int reqs = 10;

var totalTimes = new long[n];

Parallel.For(0, n, i =>
    {
        for (int req = 0; req < reqs; req++)
        {
            Stopwatch w = new Stopwatch();
            try
            {
                w.Start();

                HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://localhost:42838/Default.aspx");
                webRequest.AllowAutoRedirect = false;
                HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();

                w.Stop();
                totalTimes[i] += w.ElapsedMilliseconds;


                //Returns "MovedPermanently", not 301 which is what I want.            
                int i_goodResponse = (int)response.StatusCode;
                string s_goodResponse = response.StatusCode.ToString();
                Console.WriteLine("Normal Response: " + i_goodResponse + " " + s_goodResponse);
            }
            catch (WebException we)
            {
                w.Stop();
                totalTimes[i] += w.ElapsedMilliseconds;

                int i_badResponse = (int)((HttpWebResponse)we.Response).StatusCode;
                string s_badResponse = ((HttpWebResponse)we.Response).StatusCode.ToString();
                Console.WriteLine("Error Response: " + i_badResponse + " " + s_badResponse);
            }
        }
    });

var grandTotalTime = totalTimes.Sum();
var reqsPerSec = (double)(n * reqs * 1000) / (double)grandTotalTime;

Console.WriteLine("Requests per second: {0}", reqsPerSec);

The TPL is very useful here, as it abstracts away the detail of creating multiple threads of exececution within your process, and running each parallel branch on these threads.

Note that you still have to be careful here - we cannot share state which is updated during the tasks between threads, hence the array for totalTimes which collates the totals for each parallel branch, and only summed up at the very end, once the parallel execution is complete. If we didn't do this, we are open to the possibility of a race condition - where two seperate threads attempt to update the total count simultaneously, potentially corrupting the result.

I hope this makes sense and is useful as a start for you (I only calculate requests per second here, the other stats should be relatively easy to add). Add comments if you need further clarifications.