Trying to consume the URI cot.ag/o1LnfW from .NET with the HttpWebRequest, I get the 301 Moved response, the response header Location has a (incorrect) value of:
http://www.joycemeyer.org/BroadcastHome.aspx?video=Living_Beyond_Your_Feelings_â_Pt_1&utm_source=Twitter&utm_campaign=EEL&utm_medium=post&utm_term=September29&utm_content=post
From Fiddler, I get the (correct) Location header value:
http://www.joycemeyer.org/BroadcastHome.aspx?video=Living_Beyond_Your_Feelings_–_Pt_1&utm_source=Twitter&utm_campaign=EEL&utm_medium=post&utm_term=September29&utm_content=post
Noted the difference where the – occurs in the Fiddler URL. In the case of Fiddler, the bytes are E2 80 93. In the case of .Net, the bytes are E2 3F 3F. This results in an incorrect header interpretation, with subsequent failure to follow the redirection.
I think this is a .NET framework bug, but I have no idea what the RFCs say it SHOULD sent as. Should I report this as a bug to Microsoft, or is this a failure by bit.ly in serving the headers in the wrong code-page?
RFC 2616 specifies that the Location header should contain a URI as defined by RFC 1630, which requires a URI be 7-bit clean ASCII with any special characters URL encoded.
In other words, the server is delivering the URI incorrectly and should be escaping it.