Run x number of web requests simultaneously

Question

Our company has a web service which I want to send XML files (stored on my drive) via my own HTTPWebRequest client in C#. This already works. The web service supports 5 synchronuous requests at the same time (I get a response from the web service once the processing on the server is completed). Processing takes about 5 minutes for each request.

Throwing too many requests (> 5) results in timeouts for my client. Also, this can lead to errors on the server side and incoherent data. Making changes on the server side is not an option (from different vendor).

Right now, my Webrequest client will send the XML and wait for the response using result.AsyncWaitHandle.WaitOne();

However, this way, only one request can be processed at a time although the web service supports 5. I tried using a Backgroundworker and Threadpool but they create too many requests at same, which make them useless to me. Any suggestion, how one could solve this problem? Create my own Threadpool with exactly 5 threads? Any suggestions, how to implement this?

Answer 1

The easy way is to create 5 threads ( aside: that's an odd number! ) that consume the xml files from a BlockingCollection.

Something like:

var bc = new BlockingCollection<string>();

for ( int i = 0 ; i < 5 ; i++ )
{
    new Thread( () =>
        {
            foreach ( var xml in bc.GetConsumingEnumerable() )
            {
                // do work
            }
        }
    ).Start();
}

bc.Add( xml_1 );
bc.Add( xml_2 );
...
bc.CompleteAdding(); // threads will end when queue is exhausted