I've been searching for an answer for a while now and I am completely stumped. I am writing a screen scraper and am stuck getting a 500 Internal Error when I use the C# WebClient. I have overridden the WebClient class to include support for cookies.
On Firefox I have a plugin installed called HTTP Resource Test and by submitting the post data and correct headers with this plugin I am able to replicate exactly what I am trying to do. I think the best way right now to figure out why I can't get C# WebClient to work is to be able to see the actual request that WebClient is making to the server.
How can I view the request in string format that the WebClient is making to the web server I am trying to scrape?
You can use Fiddler2 for viewing the traffic. The easiest thing to try would be to add a user agent to the WebClient.Headers and see if this resolves your issue:
string userAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; MS-RTC LM 8)";
client.Headers[HttpRequestHeader.UserAgent] = userAgent;