I have Airflow running in a container.
I use the Docker Operator to create a number of container that run a script.
the name for the container is dynamically created with some parameters which make it meaningful to me if I list the containers to see via terminal what's going on.
For instance the container name is called my_script_daily_A
if the script runs successfully the container is removed automatically.
I have retries in case the task fails. There is a problem in case the script fails and airflow retries the run:
In case of script failure the container would still exists in exited status, it is not removed. This makes its name already assigned and therefore when Airflow runs the task again it throws an error "a container with this name already exists".
To solve this I created a script that runs on the host every minute:
for i in $(docker ps -aq --filter status=exited); do
echo 'Removing '; docker rm $i done
Im not very happy of this, I would rather trigger this script only if there is a failure, even better if I could detect whether the failure was due to the name conflict.
I know you can set up actions on call back with "on_failure_callback" attribute.
But running my script to remove the exited containers would only work if I was running Airflow directly on the host, not on the container.
Any suggestions on what to do here?
This has been already fixed in Docker Provider 1.0.1:
https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/index.html#id9
Previously when auto_remove was set to true and container errored, it was indeed not removed.
Just Upgrade to the latest Docker Provider and the problem should be fixed.