Search code examples
azureazure-devopsazure-pipelinesazure-rm-template

How to create a folder inside of an Azure Data Lake container using an ARM template?

I have an Azure ADLS storage account called eventcoadltest and I have a container called eventconnector-transformed-data-fs.

enter image description here

I have deployed this ADLS through an ARM template but I need to create a directory inside of eventconnector-transformed-data-fs as shown below (the folder debugging was created through the UI but I need to achieve the same with an ARM template):

enter image description here

I have found some posts that indicate this is not possible but it can be bypassed with some workarounds:

  1. How to create empty folder in azure blob storage
  2. Use ARM template to create directories in Azure Storage Containers?
  3. How to create a folder inside container in Azure Data Lake Storage Gen2 with the help of 'azure-storage' Package
  4. ARM template throws incorrect segments lengths for array of storage containers types
  5. how to create blob container inside Storage Account using ARM templates
  6. Microsoft Azure: How to create sub directory in a blob container
  7. How to create an azure blob storage using Arm template?
  8. How to create directories in Azure storage container without creating extra files?
  9. How to create a folder inside container in Azure Data Lake Storage Gen2 with the help of 'azure-storage' Package

I have tried to modify my ARM template as well to achieve a similar result but I haven't had any success.

{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
    "storageAccountDLName": {
        "type": "string"
    },
    "sku": {
        "type": "string"
    },
    "directoryOutput":{
        "type": "string"
    }
},
"resources": [
    {
        "type": "Microsoft.Storage/storageAccounts",
        "apiVersion": "2021-02-01",
        "sku": {
            "name": "[parameters('sku')]",
            "tier": "Standard"
        },
        "kind": "StorageV2",
        "name": "[parameters('storageAccountDLName')]",
        "location": "[resourceGroup().location]",
        "tags": {
            "Contact": "[parameters('contact')]"
        },
        "scale": null,
        "properties": {
            "isHnsEnabled": true,
            "networkAcls": {
                "bypass": "AzureServices",
                "virtualNetworkRules": [],
                "ipRules": [],
                "defaultAction": "Allow"
            }
        },
        "dependsOn": [],
        "resources": [
            {
                "type": "storageAccounts/blobServices/containers",
                "name": "[concat('default/', 'eventconnector-raw-data-fs/test')]",
                "apiVersion": "2021-02-01",
                "properties": {},
                "dependsOn": [
                    "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountDLName'))]"
                ]
            }
        ]
    }
]
}

The following code was modified for trying to create the folders inside of the containers.

"type": "storageAccounts/blobServices/containers",
"name": "[concat('default/', 'eventconnector-raw-data-fs/test')]"

The reason why I am trying to solve this problem is because I won't have access to create folders in our production environment, so that's why I need to do the deployment fully through ARM. How can I create this folder with the deployment script? Is there another alternative for achieving my desired result? Any idea or suggestion is welcome :)

Solution

  • After some research I found out that it is possible to create a folder via Databricks with the following command:

    dbutils.fs.mkdirs("dbfs:/mnt/folder_desktop/test/uploads")

    I had to configure Databricks with my Azure Datafactory in order to run this command.