I have to create a huge number of Docker container on different hosts (e.g. 50 container each on 3 hosts). These container all have the same image, configuration etc. and only the network address and ID of each container should be different (so basically I want to create a huge virtual container network). Is there a way to achieve this?
I have looked at technologies like Helios and Kubernetes but they seem to only deploy one container on each agent. I thought about just creating a lot of different jobs in Helios and then deploy each one of them to its agent, but that seems a little dirty to me.
For the Docker specific solution, you can use Swarm and Compose.
Create your Docker Swarm cluster of 3 nodes and change your environment to that Swarm. (The below assumes each host is listening on 2375, which is ok for a private network, but you'll want TLS setup and switch over to 2376 for more security.)
cat >cluster.txt <<EOF
node1:2375
node2:2375
node3:2375
EOF
docker run -d -P --restart=always --name swarm-manager \
-v ./cluster.txt:/cluster.txt \
swarm manage file:///cluster.txt
export DOCKER_HOST=$(docker port swarm-manager 2375)
Define your service inside of a docker-compose.yml, and then run docker-compose scale my-service=150. If your Swarm is setup with the default spread strategy, it will distribute them across the 3 hosts based on the number of containers running (or stopped) on each.
cat >docker-compose.yml <<EOF
my-app:
image: my-app
EOF
docker-compose scale my-app=150
Note that the downside of docker-compose over the other tools out there is that it doesn't correct for outages until you rerun it.