Storage blob delete-batch to delete all blobs excluding two directories with similar names

Question

Thanks to the answer by @Jim Xu I've gotten far, but I remain with the following issue. I want to delete all files except those in the folder data and cms.

Current files

  - cms/a2907135_1800x900.jpg
  - cms/c1964f4e_1800x900.jpg
  - css/app.9e1921c7.css
  - css/chunk-vendors.4bdbfeab.css
  - css/pageContent.88886da1.css
  - data/themes.json
  - data/website.json
  - data_old/downloads.json
  - data_old/staff.json
  - data_old/themes.json
  - data_old/website.json
  - img/flags.5fa419a7.png
  - img/star/favicon.ico
  - img/via/favicon.ico
  - img/via/favicon.png
  - img/via/mask-icon.svg
  - img/via/touch-icon-ipad-retina.png
  - img/via/touch-icon-ipad.png
  - img/via/touch-icon-iphone-retina.png
  - img/via/touch-icon-iphone.png
  - index.html
  - js/app.136c0496.js
  - js/app.136c0496.js.map
  - js/chunk-vendors.2dcffb72.js
  - js/chunk-vendors.2dcffb72.js.map
  - js/pageContent.55ebca01.js
  - js/pageContent.55ebca01.js.map
  - js/pageHome.e085832d.js
  - js/pageHome.e085832d.js.map
  - js/pageNotFound.e5d65257.js
  - js/pageNotFound.e5d65257.js.map
  - robots.txt

Using az storage blob delete-batch to delete the files with the pattern --pattern '[!data&&!cms]*' results in the following files to delete

  - img/flags.5fa419a7.png
  - img/star/favicon.ico
  - img/via/favicon.ico
  - img/via/favicon.png
  - img/via/mask-icon.svg
  - img/via/touch-icon-ipad-retina.png
  - img/via/touch-icon-ipad.png
  - img/via/touch-icon-iphone-retina.png
  - img/via/touch-icon-iphone.png
  - index.html
  - js/app.136c0496.js
  - js/app.136c0496.js.map
  - js/chunk-vendors.2dcffb72.js
  - js/chunk-vendors.2dcffb72.js.map
  - js/pageContent.55ebca01.js
  - js/pageContent.55ebca01.js.map
  - js/pageHome.e085832d.js
  - js/pageHome.e085832d.js.map
  - js/pageNotFound.e5d65257.js
  - js/pageNotFound.e5d65257.js.map
  - robots.txt

The folder data_old and css are not deleted. This is probably because of the [] used (the normally glob as or) so data_old and css are matched. But how can I delete them as well? I've tried --pattern '(!data&&!cms)*' and --pattern '!data&&!cms*' without any success.

Answer 1

That might be a hard task to do for az storage blob delete-batch. The supported patterns are limited to '*', '?', '[seq]', and '[!seq]', which are found at Pythons fnmatch module, and might not be enough for your usecase.

Instead, you could write an Azure PowerShell script to do this for you. There are alot of handy Az.Storage management cmdlets you can use to do this task.

Demo:

$storageAccountName = "STORAGE_ACCOUNT_NAME"
$storageAccountKey = "STORAGE_ACCOUNT_KEY"

$container = "web"
$foldersToKeep = "data", "cms"

$storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey

Get-AzStorageBlob -Context $storageContext -Container $container -Blob * | ForEach-Object {
    $contents = $_.Name.Split("/")
    if ($contents.Length -gt 1 -and $contents[0] -notin $foldersToKeep) {
        Write-Host "Deleting $($_.Name)"
        Remove-AzStorageBlob -Container $container -Blob $_.Name -Context $storageContext
    }
}

Which you can also wrap into a function for reusability:

function Remove-AzureBlobsFromFolders {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$true)]
        [string]$Container,
        [Parameter(Mandatory=$true)]
        [string[]]$ExcludeFolders,
        [Parameter(Mandatory=$true)]
        [string]$StorageAccountName,
        [Parameter(Mandatory=$true)]
        [string]$StorageAccountKey
    )

    $storageContext = New-AzStorageContext `
        -StorageAccountName $StorageAccountName `
        -StorageAccountKey $StorageAccountKey

    Get-AzStorageBlob -Context $storageContext -Container $Container -Blob * | ForEach-Object {
        $contents = $_.Name.Split("/")
        if ($contents.Length -gt 1 -and $contents[0] -notin $ExcludeFolders) {
            Write-Host "Deleting $($_.Name)"
            Remove-AzStorageBlob -Container $Container -Blob $_.Name -Context $storageContext
        }
    }
}

Remove-AzureBlobsFromFolders `
    -Container "web" `
    -ExcludeFolders "data", "cms" `
    -StorageAccountName "STORAGE_ACCOUNT_NAME" `
    -StorageAccountKey "STORAGE_ACCOUNT_KEY"