Every time I submit a batch job, does a new Docker container get created or the old container will be reused.
If a new Docker container is created every time, what happens to the container when the job is done.
In AWS ECS, ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION variable sets the time duration to wait from when a task is stopped until the Docker container is removed(by default 3 hours)
If all these containers only get cleanup after three hours, wouldn't the ECS container instance get filled up quick easily if I submit a lot of jobs?
Getting this error CannotCreateContainerError: API error (500): devmapper when running a batch job. Does it help if I clean up the docker container files at the end of the job?
Every time I submit a batch job, does a new Docker container get created or the old container will be reused.
Yes. Each job run on Batch will be run as a new ECS Task, meaning a new container for each job.
If all these containers only get cleanup after three hours, wouldn't the ECS container instance get filled up quick easily if I submit a lot of jobs?
This all depends on your job workloads, lengths, of jobs, disk usage, etc. With large quantities of short jobs that consume disk, this is entirely possible.
CannotCreateContainerError: API error (500): devmapper
Documentation for this error indicates a few possible solutions, however the first, which you've already called out may not help in this case.
ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION which defaults to 3h on ECS, seems to be set to 2m by default on Batch Clusters - you can inspect the EC2 User Data on one of your batch instances to validate that it is set this way on your clusters. Depending on the age of the cluster, these settings may change. Batch does not automatically update to the latest ECS Optimized AMI without creation of a whole new cluster, so I would not be surprised if it does not change settings either.
If your cleanup duration setting is currently set low, you might try creating a custom AMI which provisions a larger than normal docker volume. By default, the ECS optimized AMIs ship with an 8GB root drive, and 22GB volume for docker.