How to programmtically get a web page hosted in Sharepoint (returns Forbidden 403)

Question

I am on an internal network and I am authroised to view a web page hosted by Sharepoint - I can view the page in a browser perfectly fine.

I want to be able to 'get' the page programmatically, how can this be done?

I have tried using HttpClient GetAysnc with UseDefaultCredentials=true but it returns 403 Forbidden.

I suspect I am not appreciating the nuances of Sharepioint.

var httpClientHandler = new HttpClientHandler() {
       UseDefaultCredentials = true,
       UseProxy = true
    };
    using var httpClient = new HttpClient(httpClientHandler);
    
    var getAsyncTask = Task.Run(() => httpClient.GetAsync(match.Value, HttpCompletionOption.ResponseContentRead, cancellationToken), cancellationToken);
    getAsyncTask.Wait(cancellationToken);
    
    var response = getAsyncTask.Result;
    if (response.StatusCode == HttpStatusCode.OK)
    {
        var streamTask = Task.Run(() => response.Content.ReadAsStreamAsync(), cancellationToken);
        streamTask.Wait(cancellationToken);
    
        var stream = streamTask.Result;
    }

Answer 1

You will need to use a bearer token in the HttpClient header instead of default credentials. You can obtain the bearer token using MSAL. The first attempt at getting the token would normally be silent like this:

var accounts = await app.GetAccountsAsync();
AuthenticationResult result = null;
result = await app.AcquireTokenSilent(scopes, accounts.FirstOrDefault()).ExecuteAsync();

Of course this call could fail in which case you'd have to get authentication through user interaction like this:

result = await app.AcquireTokenInteractive(scopes).ExecuteAsync();

A complete method would look something like this:

var accounts = await app.GetAccountsAsync();
AuthenticationResult result = null;
try
{
    result = await app.AcquireTokenSilent(scopes, accounts.FirstOrDefault()).ExecuteAsync();
}
catch (MsalUiRequiredException ex)
{
    try
    {
        result = await app.AcquireTokenInteractive(scopes).ExecuteAsync();
    }
    catch (MsalException msalex)
    {
        ResultText.Text = $"Error Acquiring Token:{System.Environment.NewLine}{msalex}";
    }
}
catch (Exception ex)
{
    ResultText.Text = $"Error Acquiring Token Silently:{System.Environment.NewLine}{ex}";
    return;
}
if (result != null)
{
    //Use the token
    HttpClient httpClient = new HttpClient();
    HttpClient.DefaultRequestHeaders.Authorization =
        new AuthenticationHeaderValue("bearer", result.AccessToken);
    //Set the client to accept JSON.
    HttpClient.DefaultRequestHeaders.Accept.Add(
        new MediaTypeWithQualityHeaderValue("application/json"));
}