I am trying to download zip data from a url (https://data.police.uk/data/archive/2019-10.zip) and place into a Storage Account (blob).
The PS need to run within an Azure automation account.
When using the below, I get an exception
There is not enough space on the disk
$url = "https://data.police.uk/data/archive/2019-10.zip"
$output = "\policedata.zip"
$wc = New-Object System.Net.WebClient
$download = $wc.DownloadFile($url, $output)
Is there a way to move the zip file to an Azure storage blob within downloading it locally first, or extracting the CSV's out of it?
Azure Automation is not the right solution here given the file you are downloading is 1.6 GB. The problem is the limits Azure Automation has:
Maximum amount of disk space allowed per sandbox: 1 GB
Maximum amount of memory given to a sandbox: 400 MB
You can neither download the file before uploading nor hold it in memory and then pipe it to the storage account.
You could try to setup a two way streaming where one stream downloads file and another stream uploads the file. But this will be an overkill for what you are trying to do. I would suggest using vanilla container or VM to automate this.