Search code examples
azureazure-data-factoryazure-files

Azure Pipeline Data flow ERROR This operation is not permitted on a non-empty directory

I have issue last few days on azure factory, last week there are no issue to this error and there are no changing on ADF(Azure Data Factory) pipeline, but last 3 days this error occur

{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'savetocurated': org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.","Details":"org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.\n\tat org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2607)\n\tat org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2617)\n\tat org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteFile(NativeAzureFileSystem.java:2657)\n\tat org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.execute(NativeAzureFileSystem.java:2391)\n\tat org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor.executeParallel(AzureFileSystemThreadPoolExecutor.java:223)\n\tat org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteWithoutAuth(NativeAzureFileSystem.java:2403)\n\tat org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:2453)\n\tat org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1936)\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter."}

enter image description here

And i have tried some fix based on internet/stackoverflow

  1. Create New Linked Service = Results Still Failed
  2. Delete all data in Sink folder = Results Still Failed
  3. Copy a last success file to today error file and rerun = Results Still Failed

Maybe any info about how to handle this error?

Solution

  • I was able to proceed past this error by creating new Linked Services of Type Azure Data Lake Storage Gen2 and updating my sink datasets to use that. They were previously of type Azure Blob Storage. After this update and publishing the change my pipeline worked.