Azure Container Apps Run multiple Job with different env variable

Question

I'm new to ACA, I'm migrating a tool I made in my python API that could basically create Jobs with the k8s python SDK (example here). I would create a new job with the exact same image and configuration but with different env variable so that the script inside the Job would work on a specific scope.

I'm trying to do this with ACA python SDK, the documentation and examples are quite hard to find. But I managed to create a Job definition and trigger it multiple times and in parallel.

But I have an issue, the env variables are the same, and I don't know what to do. The image docker pull is VERY slow (10 minutes pull). So I did not want to create multiple exact same job with different env variable. I try to update my existing job definition to update the env variable. The issue is that if my API has to create 10 jobs in the same time (async) then it will upadate 10 times the Job definition, trigger 10 times the execution but with only the latest version of the update. So basically with the same env variable.

Do you know a way, a pattern, a trick that could help me do this in ACA?

Here is what I which to do:

Create jobs without repulling the Docker Image B, and with different Env Variables. I also which to restart failed jobs.

To give some context, I can not move my registery, I which i could use ACR for this, but I am not allowed.

Answer 1

I found out it was possible to define a Container job with some values, and trigger the execution job with an other template. I find this way of doing a bit weird. But it works.

from azure.identity import DefaultAzureCredential
from azure.mgmt.appcontainers import ContainerAppsAPIClient
client = ContainerAppsAPIClient(credential=DefaultAzureCredential(), subscription_id="xxxxxxxxxxxxxx")

job_parameters = {
    "location": "West Europe", # Change your location
    "properties": {
        "environmentId": "/subscriptions/xxxxxxxxxxxx/resourceGroups/xxxxxxxxxx/providers/Microsoft.App/managedEnvironments/xxxxxxxxxxxx",
        "workloadProfileName": "Consumption",
        "configuration": {
            "secrets": [
                {
                    "name": "xxxxx", # Storing my registry password
                    "value": "xxxx"
                }
            ],
            "triggerType": "Manual",
            "replicaTimeout": 1800,
            "replicaRetryLimit": 0,
            "manualTriggerConfig": {
                "replicaCompletionCount": 1,
                "parallelism": 1
            },
            "registries": [
                {
                    "server": "artifacts.xxxxxxx.com",
                    "username": "xxxxxx",
                    "passwordSecretRef": "xxx",
                },
            ]
        },
        "template": {
            "containers": [
                {
                    "image": "artifacts.xxxxxxx.com/......./my-docker-image",
                    "name": "my_job_name",
                    "env": [
                        {
                            "name": "MY_VAR_ENV",
                            "value": "toto"
                        }
                    ],
                    "resources": {
                        "cpu": 0.25,
                        "memory": "0.5Gi"
                    },
                    "command": [
                        "echo", "'Hello World'"
                    ]
                }
            ]
        }
    }
}

response = client.jobs.begin_create_or_update("my-ressource-group", "my_job_name", job_parameters).result() # This will only make a definition of your Job, it will not run it.

# Here we ca create 2 templates from the same Job with different values

for i in range(2):
    val = f"test-{i}"
    job_execution_template = {
        "containers" : [
            {
                "image": "you-can-change-your-docker-image-or-keep-the-same",
                "name": val, # This does not seem to work, it keeps the default name of the Job
                "env": [ # Can alter all env variables
                    {
                        "name": "CONTAINER_APP_JOB_NAME",
                        "value": val
                    }
                ],
                "resources": { #You can change this also
                    "cpu": 0.5,
                    "memory": "1Gi",
                },
                "command": [
                    "/bin/bash", "-c", f"echo {val}; env"
                ]
            }
        ]
    }
    # Execute the job
    response = client.jobs.begin_start(resource_group_name="my-ressource-group", job_name="my-job-name", template=job_execution_template).result()
    print(f"Job execution started with ID: {response.id}")