Strict ordering of HTTP headers in HttpWebrequest

Question

In spite of the RFC stating that the order of uniquely-named headers shouldn't matter, the website I'm sending this request to does implement a check on the order of headers.

This works:

GET https://www.thewebsite.com HTTP/1.1
Host: www.thewebsite.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 etc

This doesn't work:

GET https://www.thewebsite.com HTTP/1.1
Accept: */*
User-Agent: Mozilla/5.0 etc
Host: www.thewebsite.com
Connection: keep-alive

The default HttpWebRequest seems to put the Host and Connection headers at the end, before the blank line, rather than just after the url.

Is there any way (using a fork of HttpWebRequest or some other library in Nuget even) to specify the order of headers in a HttpWebRequest?

If possible, I'd rather not start going down the route of implementing a proxy to sort them or having to code the whole thing up using a TcpClient.

I'd appreciate any hints at all on this.

Update: With Fiddler running, header order in HttpWebrequest can be re-shuffled in CustomRules.cs. Still no closer to a solution without a proxy though.

Answer 1

.Net Core

If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:

using System.Net;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            var request = WebRequest.Create("http://www.google.com");
            request.Headers.Add("Host", "www.google.com");
            // this will be set within GetResponse.
            request.Headers.Add("Connection", "");
            request.Headers.Add("Accept", "*/*");
            request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
            request.GetResponse();
        }
    }
}

Here is an example with HttpClient:

using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp3
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var client = new HttpClient();
            client.DefaultRequestHeaders.Add("Host", "www.google.com");
            client.DefaultRequestHeaders.Add("Connection", "keep-alive");
            client.DefaultRequestHeaders.Add("Accept", "*/*");
            client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
            await client.GetAsync("http://www.google.com");
            await client.PostAsync("http://www.google.com", new StringContent(""));
        }
    }
}

Edit The above code did not work on .Net Framework only .Net Core

.Net Framework

On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.

One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.

Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize. See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079

So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.

WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES

If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework

using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            // WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
            // If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
            var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
            var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
            var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
            {
                ["Host"] = "www.google.com",
                ["Connection"] = "keep-alive",
                ["Accept"] = "*/*",
                ["User-Agent"] = "Mozilla/5.0 etc"
            });
            field.SetValue(request, headers);
            request.GetResponse();
        }
    }

    internal class CustomWebHeaderCollection : WebHeaderCollection
    {
        private readonly Dictionary<string, string> _customHeaders;

        public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
        {
            _customHeaders = customHeaders;
        }

        public override string ToString()
        {
            // Could call base.ToString() split on Newline and sort as needed

            var lines = _customHeaders
                .Select(kvp => $"{kvp.Key}: {kvp.Value}")
                // These two new lines are needed after the HTTP header
                .Concat(new [] { string.Empty, string.Empty });

            var headers = string.Join("\r\n", lines);

            return headers;
        }
    }
}