HttpClient doesn't load the content of the valid link

Question

I use HttpClient to get the content of the pages on the internet and was faced with weird behavior.

Some sites load perfectly, but some requests fail by timeout. The thing is that the links work perfectly in the browser. E.g. I have the following link: https://www.luisaviaroma.com/en-gb/shop/women/shoes?lvrid=_gw_i4. I can open it in the browser, but my code doesn't work:

var httpClient = new HttpClient();
var response = await httpClient.GetAsync("https://www.luisaviaroma.com/en-us/sw/women?lvrid=_gw");

What can be the cause of it? Probably the issue is with the _ symbol? How should I fix it then?

I also tried to use 3rd party libraries like RestSharp but got the same result.

The exception is:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Suits.Scheduler.SchedulerHostedService.UpdateClothesLinksAsync(SuitsDbContext dbContext) in C:\Repo\server\src\Suits.Scheduler\SchedulerHostedService.cs:line 98

Answer 1

I got a 403 Error, when not setting appropriate values for the Header-Accept field.
The result is a text/html, so you need to add the appropriate Header:

HttpRequestMessage msg = new HttpRequestMessage(
  HttpMethod.Get,
  "https://www.luisaviaroma.com/en-gb/shop/women/shoes?lvrid=_gw_i4"
);
msg.Headers.Add("Accept", "text/html");
HttpClient client = new HttpClient();
var response = client.SendAsync(msg).Result;

EDIT:
In the given case, OP needs to add the Accept-Encoding header two. The answer of D A pointed this out. Code to add the field:

msg.Headers.Add("Accept-Encoding", "br");