After consulting the Microsoft Support I am able to kick off a Docker container via Azure Automation with the following code:
$connection = Get-AutomationConnection -Name AzureRunAsConnection
$secpasswd = ConvertTo-SecureString "132asdf9asdf342" -AsPlainText -Force
$credentials = New-Object System.Management.Automation.PSCredential ("somecontainerregistry", $secpasswd)
Connect-AzureRmAccount -ServicePrincipal -Tenant $connection.TenantID -ApplicationId $connection.ApplicationID -CertificateThumbprint $connection.CertificateThumbprint
New-AzureRmContainerGroup -RegistryCredential $credentials -ResourceGroupName automation-rg -Name jjcontainer03 -Image somecontainerregistry.azurecr.io/etl-pipeline -OsType Linux -DnsNameLabel aci-etl-pipeline-01 -RestartPolicy Never -Command "scrapy crawl data"
This seems to work fine when I test the pane inside the Azure Automation Portal, but when I schedule it to run every hour, I only see that the Runbook job has been executed (at the correct time), without creating a new Azure instance. Should I remove the old instance every time or is there something else I am missing?
If you are specifying the script to create a container with a static name - such as the one in your case - it will not be recreated since AzureRM module detects that the said container group already exists. Try adding 'Remove-AzureRmContainerGroup ...' one line above the 'New-AzureRmContainerGroup ...'