What specific syntax must be used in an ARM template given below in order to connect a Synapse workspace to an Azure storage account and file system that are defined in a different resource group?
The fastest and easiest way to answer this question is to put the two templates given below into your own GitHub repository and to tinker with the synapse template in a destination resource group until it successfully connects the Synapse workspace to the storage account defined in the other template that you deploy into a source resource group. Then just post a link to your GitHub repository with the versions of the minimal templates that you got to work.
My question asks for this in ARM templates, using the two minimal templates given below. But if this problem turns out to be a Microsoft Bug under the hood of the ARM template engine, then a terraform version of this very simple functionality would be second best for this specific requirement.
FORMAT REQUIRED:
The resource id of the storage group that needs to be connected takes the form of:
"/subscriptions/valid-subscription-id/resourceGroups/differentresourcegroupname/providers/Microsoft.Storage/storageAccounts/nameofstorageaccount"
ERROR MESSAGE:
The error message we are getting is:
C:\Users\me>az deployment group create --name mydeploymentname --resource-group newresourcegroup --template-file C:\\path\\to\\dir\\synapse.json --verbose --parameters C:\\path\\to\\dir\\keys.json
{"code": "InvalidTemplate", "message": "Deployment template validation failed: 'The template reference 'nameofstorageaccount' is not valid: could not find template resource or resource copy with this name. Please see https://aka.ms/arm-function-reference for usage details.'.", "additionalInfo": [{"type": "TemplateViolation", "info": {"lineNumber": 0, "linePosition": 0, "path": ""}}]}
Command ran in 2.649 seconds (init: 0.392, invoke: 2.257)
LOCATION OF PROBLEM:
The part of the ARM template that defines the incorrect syntax is:
"variables": {
...
"dlsName": "[toLower(concat('dls',parameters('companyTla'),parameters('deploymentType')))]",
"dlsFsName": "[toLower(concat(variables('dlsName'),'fs1'))]",
...
}
Specifically, how do we need to change the dlsName and dlsFsName variable definitions in the preceding block in order to correctly reference the storage account and file system in the other resource group without breaking the ARM template?
RESOURCE SECTION WITH PROBLEM:
The section of the Microsoft.Synapse/workspaces resource definition that needs to ingest a correctly formed storage account and file system is:
"defaultDataLakeStorage": {
"accountUrl": "[reference(variables('dlsName')).primaryEndpoints.dfs]",
"filesystem": "[variables('dlsFsName')]"
},
COMPLETE MINIMAL ARM TEMPLATE:
The complete destination ARM template that needs to successfully receive and use the storage coordinates is:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for your deployment."
}
},
"companyTla": {
"type": "string",
"metadata": {
"description": "This is a Three Letter Acronym for your company name. 'CON' for Contoso for example."
}
},
"allowAllConnections": {
"type": "string",
"defaultValue": "true"
},
"deploymentType": {
"type": "string",
"defaultValue": "poc"
},
"sqlAdministratorLogin": {
"type": "string",
"metadata": {
"description": "The username of the SQL Administrator"
}
},
"sqlAdministratorLoginPassword": {
"type": "securestring",
"metadata": {
"description": "The password for the SQL Administrator"
}
}
},
"variables": {
"synapseName": "[toLower(concat(parameters('companyTla'),parameters('deploymentType')))]",
"dlsName": "[toLower(concat('dls',parameters('companyTla'),parameters('deploymentType')))]",
"dlsFsName": "[toLower(concat(variables('dlsName'),'fs1'))]",
"workspaceName": "[toLower(concat(variables('synapseName'),'ws1'))]"
},
"resources": [
{
"type": "Microsoft.Synapse/workspaces",
"apiVersion": "2019-06-01-preview",
"name": "[variables('workspaceName')]",
"location": "[parameters('location')]",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"defaultDataLakeStorage": {
"accountUrl": "[reference(variables('dlsName')).primaryEndpoints.dfs]",
"filesystem": "[variables('dlsFsName')]"
},
"sqlAdministratorLogin": "[parameters('sqlAdministratorLogin')]",
"sqlAdministratorLoginPassword": "[parameters('sqlAdministratorLoginPassword')]",
"managedVirtualNetwork": "default"
},
"resources": [
{
"condition": "[equals(parameters('allowAllConnections'),'true')]",
"type": "firewallrules",
"apiVersion": "2019-06-01-preview",
"name": "allowAll",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "255.255.255.255"
}
},
{
"type": "firewallrules",
"apiVersion": "2019-06-01-preview",
"name": "AllowAllWindowsAzureIps",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "0.0.0.0"
}
},
{
"type": "managedIdentitySqlControlSettings",
"apiVersion": "2019-06-01-preview",
"name": "default",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"grantSqlControlToManagedIdentity": {
"desiredState": "Enabled"
}
}
}
]
}
]
}
SOURCE OF STORAGE ACCOUNT AND STORAGE CONTAINER VALUES:
The valid values for the storage account and storage container need to originate from a different ARM template that creates the storage account and container in a different resource group, and that output their coordinates via output variables.
The current syntax for the output section of the separate storage origin ARM template is as follows:
"outputs": {
"storageAccountEndpoint": {
"type": "string",
"value": "[reference(variables('dlsName')).primaryEndpoints.Web]"
},
"storageContainer": {
"type": "string",
"value": "[concat('default/', variables('dlsFsName'))]"
}
}
The complete minimal separate storage origin ARM template is as follows:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for your deployment."
}
},
"companyTla": {
"type": "string",
"metadata": {
"description": "This is a Three Letter Acronym for your company name. 'CON' for Contoso for example."
}
},
"deploymentType": {
"type": "string",
"defaultValue": "poc",
"allowedValues": [
"devtest",
"poc",
"prod",
"shared"
],
"metadata": {
"description": "Specify deployment type: DevTest, POC, Prod, Shared. This will also be used in the naming convention."
}
}
},
"variables": {
"dlsName": "[toLower(concat('dls',parameters('companyTla'),parameters('deploymentType')))]",
"dlsFsName": "[toLower(concat(variables('dlsName'),'fs1'))]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2019-06-01",
"name": "[variables('dlsName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard_LRS"
},
"kind": "StorageV2",
"properties": {
"accessTier": "Hot",
"supportsHttpsTrafficOnly": true,
"isHnsEnabled": true
},
"resources": [
{
"name": "[concat('default/', variables('dlsFsName'))]",
"type": "blobServices/containers",
"apiVersion": "2019-06-01",
"dependsOn": [
"[variables('dlsName')]"
],
"properties": {
"publicAccess": "None"
}
}
]
}
],
"outputs": {
"storageAccountEndpoint": {
"type": "string",
"value": "[reference(variables('dlsName')).primaryEndpoints.Web]"
},
"storageContainer": {
"type": "string",
"value": "[concat('default/', variables('dlsFsName'))]"
}
}
}
I am seeking a complete example that first outputs the storage coordinates in correct form from the origin ARM template and then successfully imports the same storage coordinates into the destination ARM template.
In the synapse arm template, you just need to add the storage account resource group as a parameter:
"parameters": {
...
"dlsResourceGroupName": {
"type": "string"
},
...
},
Then create a new variable that will hold the resourceId:
"variables": {
...
"dlsResourceId": "[resourceId(parameters('dlsResourceGroupName'), 'Microsoft.Storage/storageAccounts', variables('dlsName'))]",
...
},
Then use the reference to the storage account to get the dfs endpoint:
"defaultDataLakeStorage": {
"accountUrl": "[reference(variables('dlsResourceId'), '2023-01-01').primaryEndpoints.dfs]",
...
},
Here are the two related ARM:
az deployment group create --resource-group rg1 --template-file .\storage.json
storage.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for your deployment."
}
},
"companyTla": {
"type": "string",
"defaultValue": "tla",
"metadata": {
"description": "This is a Three Letter Acronym for your company name. 'CON' for Contoso for example."
}
},
"deploymentType": {
"type": "string",
"defaultValue": "poc",
"allowedValues": [
"devtest",
"poc",
"prod",
"shared"
],
"metadata": {
"description": "Specify deployment type: DevTest, POC, Prod, Shared. This will also be used in the naming convention."
}
}
},
"variables": {
"dlsName": "[toLower(concat('dls',parameters('companyTla'),parameters('deploymentType')))]",
"dlsFsName": "[toLower(concat(variables('dlsName'),'fs1'))]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2019-06-01",
"name": "[variables('dlsName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard_LRS"
},
"kind": "StorageV2",
"properties": {
"accessTier": "Hot",
"supportsHttpsTrafficOnly": true,
"isHnsEnabled": true
},
"resources": [
{
"name": "[concat('default/', variables('dlsFsName'))]",
"type": "blobServices/containers",
"apiVersion": "2019-06-01",
"dependsOn": [
"[variables('dlsName')]"
],
"properties": {
"publicAccess": "None"
}
}
]
}
]
}
az deployment group create --resource-group rg2 --template-file .\synapse.json
synapse.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"dlsResourceGroupName": {
"type": "string",
"defaultValue": "rg1"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for your deployment."
}
},
"companyTla": {
"type": "string",
"defaultValue": "tla",
"metadata": {
"description": "This is a Three Letter Acronym for your company name. 'CON' for Contoso for example."
}
},
"allowAllConnections": {
"type": "string",
"defaultValue": "true"
},
"deploymentType": {
"type": "string",
"defaultValue": "poc"
},
"sqlAdministratorLogin": {
"type": "string",
"defaultValue": "sqladminuser",
"metadata": {
"description": "The username of the SQL Administrator"
}
},
"sqlAdministratorLoginPassword": {
"type": "securestring",
"defaultValue": "$ql@dmInUs3r",
"metadata": {
"description": "The password for the SQL Administrator"
}
}
},
"variables": {
"synapseName": "[toLower(concat(parameters('companyTla'),parameters('deploymentType')))]",
"dlsName": "[toLower(concat('dls',parameters('companyTla'),parameters('deploymentType')))]",
"dlsResourceId": "[resourceId(parameters('dlsResourceGroupName'), 'Microsoft.Storage/storageAccounts', variables('dlsName'))]",
"dlsFsName": "[toLower(concat(variables('dlsName'),'fs1'))]",
"workspaceName": "[toLower(concat(variables('synapseName'),'ws1'))]"
},
"resources": [
{
"type": "Microsoft.Synapse/workspaces",
"apiVersion": "2019-06-01-preview",
"name": "[variables('workspaceName')]",
"location": "[parameters('location')]",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"defaultDataLakeStorage": {
"accountUrl": "[reference(variables('dlsResourceId'), '2023-01-01').primaryEndpoints.dfs]",
"filesystem": "[variables('dlsFsName')]"
},
"sqlAdministratorLogin": "[parameters('sqlAdministratorLogin')]",
"sqlAdministratorLoginPassword": "[parameters('sqlAdministratorLoginPassword')]",
"managedVirtualNetwork": "default"
},
"resources": [
{
"condition": "[equals(parameters('allowAllConnections'),'true')]",
"type": "firewallrules",
"apiVersion": "2019-06-01-preview",
"name": "allowAll",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "255.255.255.255"
}
},
{
"type": "firewallrules",
"apiVersion": "2019-06-01-preview",
"name": "AllowAllWindowsAzureIps",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "0.0.0.0"
}
},
{
"type": "managedIdentitySqlControlSettings",
"apiVersion": "2019-06-01-preview",
"name": "default",
"location": "[parameters('location')]",
"dependsOn": [ "[variables('workspaceName')]" ],
"properties": {
"grantSqlControlToManagedIdentity": {
"desiredState": "Enabled"
}
}
}
]
}
]
}