How do I get HttpWebRequest.GetResponse() to time out more slowly?

Question

I'm dealing with an agitating server that takes a long time to send a response when I make a POST request. I can't use .NET 4.5, so I've been using HttpWebRequest and, more recently, WebClient. The server accepts some data via POST, takes a long time (usually 5-7 minutes) to process it, then generates a response to indicate the results.

What I'm seeing indicates something in .NET has a timeout of roughly 2 minutes on WebRequest.GetResponse(), and nothing I do seems to affect this:

• The first version of the code was completely synchronous. I set the HttpWebRequest's Timeout and ReadWriteTimeout to 10m, but after 2 minutes a WebException with a Status of "ReceiveFailure" was thrown.
• I tested out a version using BeginGetResponse() instead, but ran into a similar failure. I also noticed I wasn't supposed to be using synchronous GetRequestStream() on the same request, so it represents some fairly big changes as I ultimately want this API to look synchronous.
• I tried using WebClient.UploadStringAsync(), but it has similar issues. Right at 2 minutes, it throws a WebException (though this time with ConnectionClosed).

I've tried mucking about with properties like ServicePointManager.MaxServicePointIdleTime and it didn't change anything. I've tried using HttpWebRequest.KeepAlive. I've read several similar questions on SO but none seem to suggest anything that works for my situation. It feels like there's some timeout I can't control getting in the way, but it also feels like there should be a way to configure it. What am I missing? Here's a rough example of the synchronous version of my code, there's nothing special here:

var request = (HttpWebRequest)WebRequest.Create(clientUrl);
request.ContentType = "application/json";
request.ReadWriteTimeout = DefaultTimeoutMilliseconds;  // 600,000
request.Timeout = DefaultTimeoutMilliseconds;
request.Method = method; // "POST"
using (var writer = new StreamWriter(requestStream)) {
    writer.Write(body);
}

var response = (HttpWebResponse)request.GetResponse();
// The exception happens on the line above, the code below does not execute.
string responseBody = null;
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream)) {
    responseBody = reader.ReadToEnd();
}

response.Close();

return responseBody;

Update Some of the mystery is solved, in that I didn't realize Node had a socket timeout right at 2 minutes. When I set that to 15m now a Windows instance pointed at my Node server is happy to wait long stretches of time. But the production environment is running on Linux via Mono. I'm retesting a few things that I didn't try earlier because they were false negatives against the bad Node timeout.

Answer 1

It turns out the answer was kind of gnarly.

The server was sending 303 responses to indicate it was still processing. The Location header indicated another URL that would return either another 303 or the results. This was chosen because most HTTP clients automatically follow redirects.

On the MS CLR, this worked fine. For some reason, in the specific version of the Mono CLR I'm using, the HttpWebRequest class resends a Content-Length header when it GETs the redirect. This confused either something else in the innards of Mono or the server into waiting for data that wasn't going to be sent, and after 1m the connection died.

I solved it by setting the HttpWebRequest's AllowAutoRedirect property to false and manually handling 3xx responses. It looks like maybe later versions of Mono have fixed the problem, but I don't feel like pushing the server admins to upgrade.