azure azure-data-factory azure-virtual-machine azure-rm-template

Register Self hosted integration runtime. Cannot find path hklm

I am deploying a Datafactory and the respective self hosted integration runtime with ARM. On the VM I install the register integration runtime VM with a powershell script through VM extensions. But the extension fails every time on the following error, at first I thought the VM has no internet access to download the correct scripts. But manually I can download scripts. There seems to be some confusing with the software registry while running the script.

The error:

VM has reported a failure when processing extension 'VMNAMEinstallGW'. Error message: \"Command execution finished, but failed because it returned a non-zero exit code of: '1'. The command had an error output of: 'Get-Item : Cannot find path 'HKLM:\\Software\\Microsoft\\DataTransfer\\DataManagementGateway\\ConfigurationManager' because \r\nit does not exist.\r\nAt C:\\Packages\\Plugins\\Microsoft.Compute.CustomScriptExtension\\1.10.12\\Downloads\\0\\register-gateway.ps1:86 char:15\r...' For more information, check the instance view by executing Get-AzVmssVm or Get-AzVm (https://aka.ms/GetAzVm). These commands can be executed using CloudShell (https://aka.ms/CloudShell)\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot

the register Script

param(
    [string]
    $gatewayKey
)

# init log setting
$logLoc = "$env:SystemDrive\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\"
if (! (Test-Path($logLoc))) {
    New-Item -path $logLoc -type directory -Force
}
$logPath = "$logLoc\tracelog.log"
"Start to excute gatewayInstall.ps1. `n" | Out-File $logPath

function Now-Value() {
    return (Get-Date -Format "yyyy-MM-dd HH:mm:ss")
}

function Throw-Error([string] $msg) {
    try {
        throw $msg
    } 
    catch {
        $stack = $_.ScriptStackTrace
        Trace-Log "DMDTTP is failed: $msg`nStack:`n$stack"
    }

    throw $msg
}

function Trace-Log([string] $msg) {
    $now = Now-Value
    try {
        "${now} $msg`n" | Out-File $logPath -Append
    }
    catch {
        #ignore any exception during trace
    }

}


function Run-Process([string] $process, [string] $arguments) {
    Write-Verbose "Run-Process: $process $arguments"
    
    $errorFile = "$env:tmp\tmp$pid.err"
    $outFile = "$env:tmp\tmp$pid.out"   
    "" | Out-File $outFile
    "" | Out-File $errorFile    

    $errVariable = ""

    if ([string]::IsNullOrEmpty($arguments)) {
        $proc = Start-Process -FilePath $process -Wait -Passthru -NoNewWindow `
            -RedirectStandardError $errorFile -RedirectStandardOutput $outFile -ErrorVariable errVariable
    }
    else {
        $proc = Start-Process -FilePath $process -ArgumentList $arguments -Wait -Passthru -NoNewWindow `
            -RedirectStandardError $errorFile -RedirectStandardOutput $outFile -ErrorVariable errVariable
    }
    
    $errContent = [string] (Get-Content -Path $errorFile -Delimiter "!!!DoesNotExist!!!")
    $outContent = [string] (Get-Content -Path $outFile -Delimiter "!!!DoesNotExist!!!")

    Remove-Item $errorFile
    Remove-Item $outFile

    if ($proc.ExitCode -ne 0 -or $errVariable -ne "") {     
        Throw-Error "Failed to run process: exitCode=$($proc.ExitCode), errVariable=$errVariable, errContent=$errContent, outContent=$outContent."
    }

    Trace-Log "Run-Process: ExitCode=$($proc.ExitCode), output=$outContent"

    if ([string]::IsNullOrEmpty($outContent)) {
        return $outContent
    }

    return $outContent.Trim()
}

function Get-RegistryProperty([string] $keyPath, [string] $property) {
    Trace-Log "Get-RegistryProperty: Get $property from $keyPath"
    if (! (Test-Path $keyPath)) {
        Trace-Log "Get-RegistryProperty: $keyPath does not exist"
    }

    $keyReg = Get-Item $keyPath
    if (! ($keyReg.Property -contains $property)) {
        Trace-Log "Get-RegistryProperty: $property does not exist"
        return ""
    }

    return $keyReg.GetValue($property)
}

function Get-InstalledFilePath() {
    $filePath = Get-RegistryProperty "hklm:\Software\Microsoft\DataTransfer\DataManagementGateway\ConfigurationManager" "DiacmdPath"
    if ([string]::IsNullOrEmpty($filePath)) {
        Throw-Error "Get-InstalledFilePath: Cannot find installed File Path"
    }
    Trace-Log "Gateway installation file: $filePath"

    return $filePath
}

function Register-Gateway([string] $instanceKey) {
    Trace-Log "Register Agent"
    $filePath = Get-InstalledFilePath
    Run-Process $filePath "-era 8060"
    Run-Process $filePath "-k $instanceKey"
    Trace-Log "Agent registration is successful!"
}

Register-Gateway $gatewayKey

the ARM template that deploys the VM extension

"resources": [
        {
            "type": "Microsoft.DataFactory/factories/integrationruntimes",
            "apiVersion": "2018-06-01",
            "name": "[concat(parameters('existingDataFactoryName'), '/', parameters('IntegrationRuntimeName'))]",
            "properties": {
                "type": "SelfHosted",
                "description": "Self-hosted Integration runtime created using ARM template"
            }
        },
        {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "name": "[concat(parameters('virtualMachineName'), '/' ,parameters('virtualMachineName'), 'installGW')]",
            "apiVersion": "2019-07-01",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[resourceId('Microsoft.DataFactory/factories/integrationruntimes', parameters('existingDataFactoryName'), parameters('IntegrationRuntimeName'))]"
            ],
            "properties": {
                "publisher": "Microsoft.Compute",
                "type": "CustomScriptExtension",
                "typeHandlerVersion": "1.9",
                "autoUpgradeMinorVersion": true,
                "settings": {
                    "fileUris": [
                        "[parameters('scriptURL')]"
                    ]
                },
                "protectedSettings": {
                    "commandToExecute": "[concat('powershell.exe -ExecutionPolicy Unrestricted -File register-gateway.ps1 ', listAuthKeys(resourceId('Microsoft.DataFactory/factories/integrationruntimes', parameters('existingDataFactoryName'), parameters('IntegrationRuntimeName')), '2017-09-01-preview').authKey1)]"
                }
            }
        }
    ]

Solution

I tested it on a Windows 10 VM and ADF V2 which was already present in my environment using the below code:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "existingDataFactoryName": {
      "type": "string"
    },
    "IntegrationRuntimeName": {
      "type": "string"
    },
    "virtualMachineName": {
      "type": "string"
    },
    "existingVnetLocation": {
      "type": "string"
    },
    "scriptUrl": {
      "type": "string"
    }
  },
  "variables": {
  },
  "resources": [
    {
      "type": "Microsoft.DataFactory/factories/integrationruntimes",
      "apiVersion": "2018-06-01",
      "name": "[concat(parameters('existingDataFactoryName'), '/', parameters('IntegrationRuntimeName'))]",
      "properties": {
        "type": "SelfHosted",
        "description": "Self-hosted Integration runtime created using ARM template"
      }
    },
    {
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "[concat(parameters('virtualMachineName'), '/' ,parameters('virtualMachineName'), 'installGW')]",
      "apiVersion": "2019-07-01",
      "location": "[parameters('existingVnetLocation')]",
      "tags": {
        "vmname": "[parameters('virtualMachineName')]"
      },
      "properties": {
        "publisher": "Microsoft.Compute",
        "type": "CustomScriptExtension",
        "typeHandlerVersion": "1.7",
        "autoUpgradeMinorVersion": true,
        "settings": {
          "fileUris": [
            "[parameters('scriptURL')]"
          ]
        },
        "protectedSettings": {
          "commandToExecute": "[concat('powershell.exe -ExecutionPolicy Unrestricted -File gatewayInstall.ps1 ', listAuthKeys(resourceId('Microsoft.DataFactory/factories/integrationruntimes', parameters('existingDataFactoryName'), parameters('IntegrationRuntimeName')), '2017-09-01-preview').authKey1)]"
        }
      },
      "dependsOn":[
          "[resourceId('Microsoft.DataFactory/factories/integrationruntimes', parameters('existingDataFactoryName'), parameters('IntegrationRuntimeName'))]"
      ]
    }
  ]
}

And the script I have stored in a storage account container (container access level is not private , if its private then you need to add sastoken to the url as well) which can be accessed publicly and has url "https://storageaccount.blob.core.windows.net/test/gatewayInstall.ps1"

Note:

Please make sure that ADF,VM and VNet all are in same region. For my case its 'eastus'. And also please ensure the VM is running.

Outputs:

Reference:

I have used the PowerShell script from Github Microsoft/Quickstart/template gatewayinstall.ps1