Aks acr purge command is hanging on 'Manifests for this repository would be deleted'

Question

We have an ACR container registry in Azure. In one of the repositories we have about 4,000 images. I am trying to delete this using the ACR purge functionality using the command:

PURGE_CMD="acr purge --filter '<service-name>:.*' --dry-run --ago 250d --untagged"
az acr run --cmd "$PURGE_CMD" --registry <registry-name> --timeout 3600 /dev/null

I have set the --ago setting so that it isn't finding any images to actually delete. However, when it runs it is fine for working out what tags should be deleted (which is none), but it just hangs on determining what manifests would be deleted. Eventually the command times out at the end of the timeout period.

The output is below.

Queued a run with ID: crc
Waiting for an agent...
2023/10/19 00:18:04 Alias support enabled for version >= 1.1.0, please see https://aka.ms/acr/tasks/task-aliases for more information.
2023/10/19 00:18:04 Creating Docker network: acb_default_network, driver: 'bridge'
2023/10/19 00:18:04 Successfully set up Docker network: acb_default_network
2023/10/19 00:18:04 Setting up Docker configuration...
2023/10/19 00:18:05 Successfully set up Docker configuration
2023/10/19 00:18:05 Logging in to registry: <registry>.azurecr.io
2023/10/19 00:18:09 Successfully logged into <registry>.azurecr.io
2023/10/19 00:18:09 Executing step ID: acb_step_0. Timeout(sec): 3600, Working directory: '', Network: 'acb_default_network'
2023/10/19 00:18:09 Launching container with name: acb_step_0
DRY RUN: The following output shows what WOULD be deleted if the purge command was executed. Nothing is deleted.
Tags for this repository would be deleted: <service-name>
Manifests for this repository would be deleted: <service-name>

How can I prevent this from hanging? I'm guessing this is occurring because of the size of the repository. Is there another approach that will work for deleting images from a repository of this size?

Answer 1

Instead of trying to process all 4,000 images at once, consider purging them in smaller batches. For example, use more specific filter patterns or date ranges to limit the number of images being processed in a single run

Or Try using CLI directly:

Instead of relying on the acr purge functionality, you can use the Azure CLI directly to list and delete images. This will give you more control over the process. For example:

# List images
az acr repository show-manifests --name <registry-name> --repository <service-name> | jq -r '.[] | select(.timestamp < "your-date-cutoff").digest'

# Delete images
az acr repository delete --name <registry-name> --image <service-name>@<digest>

When using the Azure CLI approach, you can also parallelize the deletion process. By breaking the list of images to delete into smaller batches and processing them concurrently (e.g., using a script), you can speed up the process.

For Example:

This script:

1. Lists all manifests (image digests) older than a specified date.
2. Splits these manifests into batches.
3. Deletes each batch in parallel.

#!/bin/bash

# Variables
REGISTRY_NAME="<registry-name>"
REPOSITORY_NAME="<service-name>"
DATE_CUTOFF="your-date-cutoff"  # format: YYYY-MM-DDTHH:mm:ssZ
BATCH_SIZE=100

# List manifests older than the specified date
manifests=$(az acr repository show-manifests --name "$REGISTRY_NAME" --repository "$REPOSITORY_NAME" | jq -r ".[] | select(.timestamp < \"$DATE_CUTOFF\").digest")

# Split the manifests into batches and delete in parallel
batch_count=0
batch=()

for manifest in $manifests; do
    batch+=("$manifest")
    ((batch_count++))

    if [ $batch_count -eq $BATCH_SIZE ]; then
        (
            for m in "${batch[@]}"; do
                az acr repository delete --name "$REGISTRY_NAME" --image "$REPOSITORY_NAME"@"$m" --yes
            done
        ) &
        batch_count=0
        batch=()
    fi
done

# Handle the last batch (if it's not empty)
if [ $batch_count -ne 0 ]; then
    (
        for m in "${batch[@]}"; do
            az acr repository delete --name "$REGISTRY_NAME" --image "$REPOSITORY_NAME"@"$m" --yes
        done
    ) &
fi

# Wait for all parallel jobs to complete
wait

Ensure you have jq installed to process the JSON output from the Azure CLI.