WebRequest not returning HTML

Question

I want to load this http://www.yellowpages.ae/categories-by-alphabet/h.html url, but it returns null

In some question I have heard about adding Cookie container but it is already there in my code.

var MainUrl = "http://www.yellowpages.ae/categories-by-alphabet/h.html";
HtmlWeb web = new HtmlWeb();
web.PreRequest += request =>
{
    request.CookieContainer = new System.Net.CookieContainer();
    return true;
};
web.CacheOnly = false;
var doc = web.Load(MainUrl);

the website opens perfectly fine in browser.

Answer 1

You need CookieCollection to get cookies and set UseCookie to true in HtmlWeb.

CookieCollection cookieCollection = null;
var web = new HtmlWeb
{
    //AutoDetectEncoding = true,
    UseCookies = true,
    CacheOnly = false,
    PreRequest = request =>
    {
        if (cookieCollection != null && cookieCollection.Count > 0)
            request.CookieContainer.Add(cookieCollection);

        return true;
    },
    PostResponse = (request, response) => { cookieCollection = response.Cookies; }
};

var doc = web.Load("https://www.google.com");