Uploading files to and Downloading files from 2FA protected Sharepoint from a C# Console App

Question

I am tearing my hair out over this one. There seem to be 1000 suggestions to do what I want, but most of them are obsolete, or don't work. The microsoft docs are worse than useless, don't explain the principles clearly, introduce a bunch of complex and obscure terms, and then before explaining anything branch off into a web of links to other pages which then do the same. Whenever I think I've found the answer, it turns out that the method I'm looking at has been or is about to be deprecated.

So I'll run down some questions here and lay out what I have working. Firstly, my requirements:

• the method has to run unattended as a daily scheduled job from a dotNet console application
• it is acceptable to need to do 2FA web based authentication once only when setting up the job (and perhaps at long intervals, like annually)
• it has to be able to upload file to or download files from a 2FA protected sharepoint site which is not under my direct control, but which I can communicate with the owner and request configuration changes be made

What I have working

Thanks to this video, I have managed to get 2FA authentication working from a console app for a Microsoft 365 email address. The Authentication part of the code looks like this:

    var scopes = new string[] { "https://graph.microsoft.com/.default" };

    var confidentialClient = ConfidentialClientApplicationBuilder
            .Create(spServiceProfile.ClientID)
            .WithClientSecret(spServiceProfile.ClientSecret)
            .WithAuthority(new Uri("https://login.microsoftonline.com/" + spServiceProfile.TenantID + "/"))
            .Build();

          var authResult = await confidentialClient
                .AcquireTokenForClient(scopes)
                .ExecuteAsync();

So we need three things: TenantID, ClientID, and ClientSecret. The video shows how to obtain or configure those.

To do a file upload or download, I need to choose the API or service I use to access Sharepoint. There seem two clear choices, CSOM and GraphServiceClient.

I have found code for GraphServiceClient (but haven't tried it out yet) that looks like:

      uploadedFile = (_graphServiceClient
      .Sites["root"]
      .Drives["{DriveId}"]
      .Items["{Id_of_Targetfolder}"]
      .ItemWithPath(fileToUpload.FileName)
      .Content.Request()
      .PutAsync<DriveItem>(ms)).Result;

This seems at first sight to be a lot less flexible than the CSOM way, which I have working code for, but which stopped authenticating when 2FA was introduced, since it just logged in using a username and password. It looks like:

            var authMgr = new PnP.Framework.AuthenticationManager(spServiceProfile.ClientID, spServiceProfile.UserName, securePassword);
            using (var rootCtx = authMgr.GetContext(rootSiteUrl)) {
                Uri webUri = Web.WebUrlFromPageUrlDirect(rootCtx, new Uri(rootSiteUrl + pathUrl));

                using (var ctx = authMgr.GetContext(webUri.AbsoluteUri)) {
                    var list = ctx.Web.GetList(pathUrl);

                    var sharepointFolders = list.RootFolder.Folders;
                    ctx.Load(sharepointFolders);
                    ctx.ExecuteQuery();

                    Folder sharepointRootFolder = null;
                    foreach (var sharepointFolder in sharepointFolders) {
                        //s1 += "\r\n\\" + folder.Name;
                        if (sharepointFolder.Name.ToLower() == sharepointJob.SharepointRootFolder.ToLower()) { 
                            sharepointRootFolder = sharepointFolder;
                        }
                    }

                    if (fileOperation == FileOperationEnum.UploadFolder) {
                        var job = (SharepointTransferFolderJobClass)sharepointJob;

                        // cycle through all files in folder
                        var f_sarr = Directory.EnumerateFiles(job.LocalRootFolder);
                        foreach (var localFilePath in f_sarr) {
                            var localFileName = Path.GetFileName(localFilePath);
                            sharepointRootFolder.UploadFile(localFileName, job.LocalRootFolder, true);
                        }

                        sharepointRootFolder.Update();
                        ctx.Load(sharepointRootFolder);
                        ctx.ExecuteQueryRetry();
                    }

                    if (fileOperation == FileOperationEnum.DownloadFolder) {
                        var job = (SharepointTransferFolderJobClass)sharepointJob;

                        foreach (var remoteFile in sharepointRootFolder.Files) {
                            var backupFileName = remoteFile.Name;
                            var sharepointFile = sharepointRootFolder.GetFile(backupFileName);
                            ctx.Load(sharepointFile);
                            ctx.ExecuteQueryRetry();

                            ClientResult<Stream> stream = sharepointFile.OpenBinaryStream();
                            ctx.ExecuteQueryRetry();

                            var destFullFilePath = Path.Combine(job.LocalRootFolder, backupFileName);
                            using (Stream fileStream = new FileStream(destFullFilePath, FileMode.Create)) {
                                CopyStream(stream.Value, fileStream);
                            }
                        }
                    }
                }
            }

OK, so we finally get to the questions:

1. Is CSOM about to be deprecated?

This is a guide to using CSOM but it has a prominent warning at the top: 'SharePoint add-in model is currently still fully supported, but will be officially deprecated soon (by end of 2023) in SharePoint Online. Consider using alternative extensibility options to be more future proven with your extensibility.'

I don't know if I am using SharePoint add-in model, or if that refers to CSOM in its entirety, and even if I am using Sharepoint Online (as opposed to on-prem, I suppose, but is it actually on-prem for anyone nowadays or is that just a licensing artefact?).

So would my use case fall under the deprecation, and overall does it look like CSOM is heading for deprecation in the longer term? Or is it part of the long term ecosystem?

2. If not slated for deprecation, how do I make CSOM work with my authentication method?

I haven't bee able to find anything on combining the two, and intellisense isn't yielding anything useful.

3. Is GraphServiceClient a better choice overall?

It seems that Graph is in its ascendant phase, so it may be a better bet in the long term. But I'm new to it and not sure what it is intended for (GraphQL supersedes REST, right?). Am I using a side feature of it that may get removed at some point in the future, or is this exactly what it's intended for?
I need to be able to do things like enumerating sharepoint folders and files and getting last modified dates. Can it do this?

Basically I need to know where CSOM and GraphServiceClient sit in the Sharepoint ecosystem. Are they different tools intended for different purposes, and will both be around for a long time? Is GraphServiceClient superseding CSOM?

Thanks for any answers! It used to be as simple as having a username and password. I have no doubt that anyone who understands the intricacies of all this could build an entire career on it at this point.

EDIT: thanks for the answer @Nikolay. Posting the comment from @Jansenbe in the referenced article in case the link dies.

To fully understand the story you need to know a bit more about our history. 8 years ago we introduced our first .Net library named PnP-Sites-Core. This was mainly a set of CSOM extensions that grew to be a very large set of features to help SharePoint devs. This library was supported for both SPO as various SP on-premises versions. Given there were more and more non CSOM calls needed + the old library was not ready for modern .NET + the fact that on-prem usage was going down we decided on building a new library, which is PnP Core SDK. Given PnP-Sites-Core was so huge we'd never be able to in parallel build a new library will all features and then ship it to replace PnP-Sites-Core, this transformation is multi year journey given this is driven as an open source project. To enable that we did introduce PnP Framework, which started of as cleaned up version of PnP-Sites-Core (all on-prem code specific code was dropped, all 'legacy' stuff was dropped). Folks could migrate to PnP Framework and in parallel we developed PnP Core SDK and moved functionality from PnP Framework to PnP Core SDK. For example the pages API logic lives in PnP Core SDK, but you can still call it via PnP Framework as that acts as a proxy.

When you're starting a new project it's advised to use PnP Core SDK unless you've a dependency on the PnP provisioning engine (for creating and applying site templates). When using PnP Core SDK you can still bump into certain missing features, but we try to respond quicker on those and you can also make custom API calls to REST/Graph to get a quick fix (see https://pnp.github.io/pnpcore/using-the-sdk/basics-customapirequests.html). The advantage of PnP Core SDK is that it abstracts the used APIs (SharePoint REST, Microsoft Graph or SharePoint CSOM) providing you with a unified developer experience + enabling the SDK to replace API implementations when more/better Microsoft Graph APIs become available.

Answer 1

1. Not as of now. The SharePoint add-in model is not directly related to CSOM. Although, CSOM can be used to build SharePoint Add-Ins.

2. You seem to be using PnP Framework. It support the interactive login (i.e. 2FA login) out of the box using CreateWithInteractiveLogin method for example:

var authMgr = PnP.Framework.AuthenticationManager
    .CreateWithInteractiveLogin(spServiceProfile.ClientID);
using (var rootCtx = authMgr.GetContext(rootSiteUrl)) {

You (or, rather, the owner of the Azure AD where the SharePoint belongs to) still needs to register your app, and give it proper permissions for the SharePoint, i.e. from the list of options to grant permissions, you need to select SharePoint and then grant something like AllSites.Read

Note that 2FA is designed to work with interactive users; i.e. you should NOT use it for service accounts. If you want unattended authentication, you should use (create) a certificate login instead (a self-signed certificate is fine), and then ask the owner of SharePoint to upload it into your app registration as a secret. Then you can authenticate with CSOM like this:

var authMgr = PnP.Framework.AuthenticationManager
    .CreateWithCertificate(spServiceProfile.ClientID, 
        "YOURPFX.pfx", "PFXPASS", "YOURTENANT");
using (var rootCtx = authMgr.GetContext(rootSiteUrl)) {

Note that if the SharePoint owner cares, he can grant your app access only to a specific site(s) when adding it (he needs to select Sites.Selected in this case, and then the sites)

A detailed instruction how to configure SharePoint can be found here on the Microsoft web site, for example.

3. The Graph SharePoint API still lacks a number of features that is provided by SharePoint REST API (i.e. CSOM). But if you don't need this advanced functionality, you can also go with graph API as well. Although, I don't think it makes much sense in your case, as your code is already written for CSOM. If you switch to certificate authentication instead of username/password, you should be up and running in five minutes with your existing solution.

I've written an article here to summarize it, maybe you will find it helpful.