HTTPWebRequest 403 Forbidden - faking web browser

Question

Here is my code faking a browser request:

HttpWebRequest webReq = WebRequest.CreateHttp(url);
webReq.CookieContainer = new CookieContainer();
webReq.Method = "GET";
webReq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0";
webReq.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
webReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;q=0.5");
webReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");

webReq.AllowAutoRedirect = true;
webReq.MaximumAutomaticRedirections = 50;

using (WebResponse response = webReq.GetResponse())
{
    using (Stream stream = response.GetResponseStream())
    {
        StreamReader reader = new StreamReader(stream);
        return reader.ReadToEnd();
    }
}

But I get a 403 HTTP error with this URL: http://www.alchourouk.com/xml_top_article/rss.xml

I don't understand why, there is no authentication, no cookies used...

In Postman, I have no error, and here is the code it generates:

var client = new RestClient("http://www.alchourouk.com/xml_top_article/rss.xml");
var request = new RestRequest(Method.GET);
request.AddHeader("postman-token", "1b40ceba-6311-0fe2-1566-62a2d59950a0");
request.AddHeader("cache-control", "no-cache");
IRestResponse response = client.Execute(request);

Nothing special here. Any idea why my code gets a 403 error?

Answer 1

What you need is the Referer HTTP header:

webReq.Referer = "http://www.alchourouk.com/";

And it'll work just fine.

Live demo here