Azure Container App Job on a event trigger executes every pooling interval instead of when a new message arrives in queue

Question

I have a container app job with an event trigger that should only execute when a new message is in a blob storage queue. However it executes on every pooling interval (i have set it to 5 minutes), which just consumes additional resources without processing anything. Below is my JSON configuration of the job:

{
    "id": "***",
    "name": "***",
    "type": "Microsoft.App/jobs",
    "location": "East US",
    "properties": {
        "provisioningState": "Succeeded",
        "environmentId": "***",
        "workloadProfileName": "Consumption",
        "configuration": {
            "secrets": [
                {
                    "name": "connection-string-secret"
                },
                {
                    "name": "***"
                }
            ],
            "triggerType": "Event",
            "replicaTimeout": 4000,
            "replicaRetryLimit": 1,
            "manualTriggerConfig": null,
            "scheduleTriggerConfig": null,
            "eventTriggerConfig": {
                "replicaCompletionCount": 1,
                "parallelism": 1,
                "scale": {
                    "minExecutions": 0,
                    "maxExecutions": 2,
                    "pollingInterval": 300,
                    "rules": [
                        {
                            "name": "queue",
                            "type": "azure-queue",
                            "metadata": {
                                "accountName": "***",
                                "queueLength": "1",
                                "queueName": "***"
                            },
                            "auth": [
                                {
                                    "secretRef": "connection-string-secret",
                                    "triggerParameter": "connection"
                                }
                            ]
                        }
                    ]
                }
            },
            "registries": [
                {
                    "server": "***.azurecr.io",
                    "username": "***",
                    "passwordSecretRef": "***"
                }
            ],
            "dapr": null
        },
        "template": {
            "containers": [
                {
                    "image": "***.azurecr.io/***:latest",
                    "name": "***_job",
                    "env": [
                        {
                            "name": "AZURE_STORAGE_QUEUE_NAME",
                            "value": "***"
                        },
                        {
                            "name": "AZURE_STORAGE_CONNECTION_STRING",
                            "secretRef": "connection-string-secret"
                        }
                    ],
                    "resources": {
                        "cpu": 4,
                        "memory": "8Gi"
                    }
                }
            ],
            "initContainers": null,
            "volumes": null
        },
        "eventStreamEndpoint": "***"
    },
    "identity": {
        "type": "None"
    }
}

Any ideas why this could happen? Thanks in advance!

Answer 1

If you find that your Azure Container App Job is executing on every polling interval instead of being triggered by a new message in a Blob Storage queue, it may be beneficial to review the configuration and logs for potential issues. It’s worth noting that if the problem persists even after reconfiguring the job, the issue may lie with the queue itself.

As you have rightly mentioned, in order to resolve this, we can try the following steps: Delete the Problematic Queue i.e. deleting the queue that seems to be causing the issue can help reset any erroneous state or Recreate the Queue by running the following command, ensuring that the connection string matches the one used in your Container App Job's connection-string-secret:

az storage queue create --name "<QUEUE_NAME>" --account-name "<STORAGE_ACCOUNT_NAME>" --connection-string "<CONNECTION_STRING>"

Now coming to the question "are there any logs that I can see if the container job cannot connect to the queue and pinpoint the exact problem that way?"

In general, Azure provides logging for services that can help pinpoint issues. For Azure Container App Jobs, we can enable logging and monitor the logs for any errors related to the queue connection:

-   Navigate to your Container App resource in the Azure portal.
-   Under "Settings", find the "Monitoring" or "Logs" section.
-   Ensure that logging is enabled and configured to capture the necessary information.

But yes, specifically readymade KQL queries might not be available for any custom logs. But you can always write a custom KQL query as per your requirement. If logging is already enabled, review the logs in Azure Monitor or Log Analytics -> Go to your Log Analytics workspace and query the logs for any errors or unusual patterns that might indicate a connection issue with the queue.

References:

• Tutorial: Deploy an event-driven job with Azure Container Apps
• Jobs in Azure Container Apps - Job trigger types