Search code examples
azureazure-storageazure-data-lake

File system creation in Azure data lake storage account through ARM template

I have created an Azure Data Lake gen 2 with ARM templates. But now I am trying to figure out how to create Data Lake File systems in ARM but can't seem to find the API's to do this. Is this unavailable, is this possible through another way ?

Tried to create the File systems manually and export the template in the portal but didn't seem to see the File system resource. Only this warning. enter image description here

Because this stated that the file system may be listen in the API as "blobservices/containers" I tried to add this resource to the ARM template

 {
            "name": "[concat( parameters('DataLakeName'), '/default/input')]",
            "type": "Microsoft.Storage/storageAccounts/blobServices/containers",
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('DataLakeName'))]"
            ],
            "apiVersion": "2018-07-01",
            "properties": {
                "publicAccess": "None",
                "metadata": {}
            },
            "resources": []
        }

But this did not work unfortunately and provided this error message : Blob API is not yet supported for hierarchical namespace accounts. Which got me thinking this may be not even possible to deploy through ARM. Is there someone who already tried this out ?

The ARM template resource block of my datalake storage account complete as context:

 {
            "name": "[parameters('DataLakeName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2018-07-01",
            "location": "[parameters('location')]",
            "tags": {},
            "properties": {
                "accessTier": "[parameters('accessTier')]",
                "supportsHttpsTrafficOnly": "[parameters('supportsHttpsTrafficOnly')]",
                "isHnsEnabled": true
            },
            "resources": [
                {
                    "type": "providers/advancedThreatProtectionSettings",
                    "name": "Microsoft.Security/current",
                    "apiVersion": "2017-08-01-preview",
                    "dependsOn": [
                        "[resourceId('Microsoft.Storage/storageAccounts/', parameters('DataLakeName'))]"
                    ],
                    "properties": {
                        "isEnabled": true
                    }
                }
            ],
            "dependsOn": [],
            "sku": {
                "name": "[parameters('accountType')]"
            },
            "kind": "StorageV2"
        }
Solution

  • I found this solution on this github issue answer. From a guy who find out that there isn't any solution yet so the basic thing to do here is to call a rest api manually to do so. Explanation and the blog can be found in this link: http://sql.pawlikowski.pro/2019/03/10/connecting-to-azure-data-lake-storage-gen2-from-powershell-using-rest-api-a-step-by-step-guide/

    Here is the Powershell script to create a File system in case the link is going to be deprecated:

    All credits to Michał Pawlikowski, Thanks for creating this script worked like a charm. 

    [CmdletBinding()]
Param(
  [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
  [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
  [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey
)

# Rest documentation:
# https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem/create

$date = [System.DateTime]::UtcNow.ToString("R") # ex: Sun, 10 Mar 2019 11:50:10 GMT

$n = "`n"
$method = "PUT"

$stringToSign = "$method$n" #VERB
$stringToSign += "$n" # Content-Encoding + "\n" +  
$stringToSign += "$n" # Content-Language + "\n" +  
$stringToSign += "$n" # Content-Length + "\n" +  
$stringToSign += "$n" # Content-MD5 + "\n" +  
$stringToSign += "$n" # Content-Type + "\n" +  
$stringToSign += "$n" # Date + "\n" +  
$stringToSign += "$n" # If-Modified-Since + "\n" +  
$stringToSign += "$n" # If-Match + "\n" +  
$stringToSign += "$n" # If-None-Match + "\n" +  
$stringToSign += "$n" # If-Unmodified-Since + "\n" +  
$stringToSign += "$n" # Range + "\n" + 
$stringToSign +=    
                    <# SECTION: CanonicalizedHeaders + "\n" #>
                    "x-ms-date:$date" + $n + 
                    "x-ms-version:2018-11-09" + $n # 
                    <# SECTION: CanonicalizedHeaders + "\n" #>

$stringToSign +=    
                    <# SECTION: CanonicalizedResource + "\n" #>
                    "/$StorageAccountName/$FilesystemName" + $n + 
                    "resource:filesystem"# 
                    <# SECTION: CanonicalizedResource + "\n" #>

$sharedKey = [System.Convert]::FromBase64String($AccessKey)
$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = $sharedKey

$signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))


$authHeader = "SharedKey ${StorageAccountName}:$signedSignature"

$headers = @{"x-ms-date"=$date} 
$headers.Add("x-ms-version","2018-11-09")
$headers.Add("Authorization",$authHeader)

$URI = "https://$StorageAccountName.dfs.core.windows.net/" + $FilesystemName + "?resource=filesystem"

Try {
    Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
}
catch {
    $ErrorMessage = $_.Exception.Message
    $StatusDescription = $_.Exception.Response.StatusDescription
    $false

    Throw $ErrorMessage + " " + $StatusDescription
}