Search code examples
c#.net-coredotnet-httpclienthttp-redirect.net-5

C# HttpClient not handling 304/307 redirects

I'm trying to get a HttpClient to pull favicons from websites, and for 99% of cases this code is working as expected:

var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromSeconds(10);

var response = await httpClient.GetAsync("https://www.tesco.com/favicon.ico");
response.EnsureSuccessStatusCode();

I've been finding with a couple of websites that my GetAsync method is just timing out, and I believe it is to do with it's redirect. If you run the above code in a console app, the following exception is thrown inside a TaskCanceledException and an IOException after 10 seconds:

SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

This request works completely fine on Postman, and I've tried https and http without success. When visiting the site and using Chrome's dev tools, it looks like if you use http rather than https, it returns a 307 and redirects you to the https site, which then returns a 304. I don't understand why the HttpClient is just timing out rather than giving a useful response.

Is there any way to get this to work? Am I missing something simple?

Update - 2021-01-29

I've tried this on multiple different .NET versions, and found that this could be a bug with HttpClient, as this code works for .NET Core 2.0 but not for .NET Core 2.1.

Versions Tested

  • .NET Framework 4.8: Works
  • .NET Core 1.0: Works
  • .NET Core 1.1: Works
  • .NET Core 2.0: Works
  • .NET Core 2.1: Does not work
  • .NET Core 2.2: Does not work
  • .NET Core 3.0: Does not work
  • .NET Core 3.1: Does not work
  • .NET 5: Does not work
Solution

  • It turns out this is not a .NET issue at all. I raised this issue on GitHub as all evidence was pointing towards the HttpClientHandler breaking after .NET Core 2.1. Stephen was able to point out to me that before .NET Core 2.1, a Connection: Keep-Alive header is set by default in the HttpClient.

    The culprit URLs that I have been testing against seem to require one of the following things to work:

    • A Connection: Keep-Alive header
    • An Accept: */* header, or something more specific to what is being requested
    • The request to be sent using the HTTP/2 protocol.

    For reference, to apply one of the Header fixes, use httpClient.DefaultRequestHeaders.Add(string, string), and to set the HTTP protocol, use httpClient.DefaultRequestVersion = HttpVersion.Version20;

    This issue turns out to not be a direct issue with C#, but that the default headers that are sent changed after .NET Core 2.1. I found the issue to be reproducible on Postman if you disable all headers (including the Postman token header).