.Net Core 3.0 while trying take html content from url This error page might contain sensitive error

Question

Like I said i trying to get some HTML content from URL but it's telling me "This error page might contain sensitive error" How can i solve it ? My code :

try
{
    string siteContent = string.Empty;
    string url = "https://www.antalyaeo.org.tr/tr/nobetci-eczaneler";
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.AutomaticDecompression = DecompressionMethods.GZip;
    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())   
    using (Stream responseStream = response.GetResponseStream())               
    using (StreamReader streamReader = new StreamReader(responseStream))        
    {
        siteContent = streamReader.ReadToEnd(); 
    }
    return siteContent;
}

catch (WebException webex)
{
    WebResponse errResp = webex.Response;
    using (Stream respStream = errResp.GetResponseStream())
    {
        StreamReader reader = new StreamReader(respStream);
        string text = reader.ReadToEnd();
        return text;

    }
}

Error message :

This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using <customErrors mode="Off"/>. Consider using <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> in production environments.-->

What I tried: When I search google and StackOverflow I saw they said make customError="False" or kinda like that thing from system.web / web.config But both of the files have on my project. How can I solve it? Thanks.

Answer 1

You need to add user-agent otherwise it seems server denies your request, for example:

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())