C# HttpClient.SendAsync throw "An error occurred while sending the request" exception when testing some URLs

Question

I am developing a C# console application for testing whether a URL is valid or not. It works well for most of URLs and can get response with HTTP Status Code from the target website. But when testing some other URLs, the application throws an exception saying "An error occurred while sending the request" when running HttpClient.SendAsync method. So I can't get any response or HTTP Status Code although this URL actually works in the browser. I am desperate to find out how to handle this case. If the URL doesn't work or the server rejects my request, it should at least give me the corresponding HTTP Status code.

Here is the simplified code of my test application:

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace TestUrl
{
    class Program
    {
        static void Main(string[] args)
        {
           // var urlTester = new UrlTester("http://www.sitename.com/wordpress"); // works well and get 404
           // var urlTester = new UrlTester("http://www.fc.edu/"); // Throw exception and the URL doesn't work
           var urlTester = new UrlTester("http://www.ntu.edu.tw/english/"); // Throw exception and the URL works actually

            Console.WriteLine("Test is started");

            Task.WhenAll(urlTester.RunTestAsync());
            
            Console.WriteLine("Test is stoped");
            Console.ReadKey();
        }


        public class UrlTester
        {
            private HttpClient _httpClient;
            private string _url;

            public UrlTester(string url)
            {
                _httpClient = new HttpClient();
                _url = url;
            }

            public async Task RunTestAsync()
            {
                var httpRequestMsg = new HttpRequestMessage(HttpMethod.Head, _url);

                try
                {
                    using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
                    {
                        Console.WriteLine("Response: {0}", response.StatusCode);
                    }
                }
                catch (Exception e) 
                { 
                }
            }
        }

    }
} 

Answer 1

If you look at the InnerException you will see that:

"The remote name could not be resolved: 'www.fc.edu'"

This URL does not work on my browser either.

In order to get an HTTP response you need the client to be able to communicate with the server (even in order to get error 404) and in your case the error occurred at the DNS level.

Some browsers have auto-completion for this kind of cases where if a specific URL is not found, the browser retries with a different suffix/prefix, for example:

try "x"
if didn't work, try "www." + x
if this didn't work try "www." + x + ".com"
if this didn't work try "www." + x + ".net"
if this didn't work try "www." + x + "." + currentRegionSuffix.

But note that you can change your code from:

catch (Exception e)
{

}

To:

catch (HttpRequestException e)
{
    Console.WriteLine(e.InnerException.Message);
}

And you will be able to see what causes your error.

Also, You should never want to catch the generic Exception unless the thrower has thrown the generic Exception, and even than, never catch and do nothing with the exception, at least log it.

Notice than since you wait only for that one task you can use:

urlTester.RunTestAsync().Wait();

Instead of:

Task.WhenAll(urlTester.RunTestAsync());

Task.WhenAll creates a new Task when the given Tasks are completed. in your case you need Task.WaitAll or Task.WhenAll(...).Wait().