Can't export Office 365 share point datasets through azure data factory

Question

I am building an azure marketplace offering that deploys and sets up an azure data factory and a storage account that is used as a sink. The data factory has a pipeline with several copy data activities. All of them export data from Office 365 such as BasicDataSet_v0.User_v1 and BasicDataSet_v0.SharePointPermissions_v1 into a storage account.

I have the marketplace offer in preview mode and have installed it on a test tenant where I enabled Data Connect organization wide about a week ago. So, all data sets should be available since it can take up to 48 hours for them to be accesibile.

Upon running the pipeline, all but the SharePoint data sets are being exported where the failing activities fail with:

Operation on target SharePointSites failed: ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. office365LoadErrorType: PermanentError. Your dataset request failed. Please reach out to [email protected] for further support. For more information please review our documentation at https://aka.ms/mgdcdocs.,Source=Microsoft.DataTransfer.ClientLibrary,'

Installing the marketplace offering on the tenant where I develop the app, all the data sets are being exported successfully. So, I am guessing that I've either enabled data connect wrongly, which seems like a stretch since is just a checkbox, or there is something else I have missed while following the data connect tutorial.

I thought that there might be a limitation with it being a managed app exporting Office 365 data, so I moved the resources from the managed resource group to a regular resource group where that variable should be eliminated. No luck. Exporting data from Office 365 through Data Factory needs all the resources involved such as the sink and the service principal used for authentication to be under the same tenant, so I made sure that's the case.

Have any of you fellow humans encountered this issue while exporting Office 365 SharePoint data sets using Azure Data Factory? Any help would be highly appreciated.

Here's an image of the failing activities.

Answer 1

I had reached out to Microsoft and there was an issue on their side where the share point datasets were not ready to be requested yet.

Graph Data Connect is still in preview so that might have been the issue. If you are reading this and you're having the same issue after 48 hours after enabling Data Connect, you might want to reach out to Microsoft as well. They most likely need to do some manual stuff to make the data sets available.