How to get/read text file from a url using C#?

Question

I'm trying to read text from URL:
https://whitworthpirates.com/services/schedule_txt.ashx?schedule=139

Here's my code:-

     var webRequest = WebRequest.Create(SourceUrl);

            var response = webRequest.GetResponse();
            var content = response.GetResponseStream();
           
             using (var reader = new StreamReader(content))
             {
                string strContent = reader.ReadToEnd();
                 HttpContext.Current.Response.Write("Read: "+strContent+"<br />");
             }
            HttpContext.Current.Response.Flush();
            HttpContext.Current.Response.End();

HttpContext.Current.Response.Write prints out nothing. I've tried the same code with https://google.com and StackOverflow webpages and it gets the text from these sites just fine.

I've also tried to download the file from the same URL using the following code:-

WebClient webClient = new WebClient();
webClient.DownloadFile(url, @"c:\myfile.txt");

It managed to download a .txt file on my server but it is an empty file with a size of 0KB.

Can someone help me with this, please?

Answer 1

It seems that website requires you to send a UserAgent header. (Seems it can be anything)

var webRequest = (HttpWebRequest)HttpWebRequest.Create("https://whitworthpirates.com/services/schedule_txt.ashx?schedule=139");
webRequest.UserAgent = "Hej";

var response = webRequest.GetResponse();
var content = response.GetResponseStream();

using (var reader = new StreamReader(content))
{
    string strContent = reader.ReadToEnd();
    strContent.Dump();
}