I am working with a scientific code which was packaged in Docker containers by another person. I am not very familiar with all the magic behind containers, images etc. And I was using it only by running some simple commands like docker-compose up or docker-compose up --build if I needed to add some dependencies in the code.
So everything was perfectly normal until last night. I was running a simulation that took a whole night, but I saw that results were not ok, so I just killed the processed by pressing ctrl + C 2 or 3 times. When I tried to launch simulation again by docker-compose up I got an error which, unfortunately, I can't recall right now. Also, strange thing - at that moment I couldn't connect to Internet. I rebooted, Internet worked fine again, I tried to run docker-compose up again and I got the following output:
WARNING: The DOCKERHOST variable is not set. Defaulting to a blank string.
Creating alcor_alcor_1 ...
alcor_cassandra_1 is up-to-date
Creating alcor_alcor_1 ... error
ERROR: for alcor_alcor_1 Cannot create container for service alcor: invalid IP address in add-host: ""
ERROR: for alcor Cannot create container for service alcor: invalid IP address in add-host: ""
ERROR: Encountered errors while bringing up the project.
I deleted all the images, containers and volumes by running:
docker rm $(docker ps -a -f status=exited -q)
docker rmi $(docker images -a -q)
docker volume rm $(docker volume ls -f dangling=true -q)
And then I rebuilt everything by running: docker-compose build --no-cache
No errors were thrown. I ran docker-compose up again and still got the same error: invalid IP address in add-host: ""
Then I repeated my steps again: deleted all images, containers and volumes, rebuilt everything and tried to run again. Now every time it shows me this:
WARNING: The DOCKERHOST variable is not set. Defaulting to a blank string.
Creating network "alcor_default" with the default driver
ERROR: Failed to program FILTER chain: iptables failed: iptables -I FORWARD -o br-231cf5f5b939 -j DOCKER: iptables v1.4.14: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
(exit status 2)
What could be the reason for this and how do I resolve it? Can I provide any additional information to help understand this? Google and Docker docs don't say anything on this matter. And I didn't find anything on SO about it. Any help is appreciated.
Here is something that probably could help understand the issue:
$ docker version
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:27:03 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:22:34 2017
OS/Arch: linux/amd64
Experimental: false
docker-compose.yml
version: '3'
services:
alcor:
build: .
image: lycantropos/alcor:latest
entrypoint: "/alcor/docker-entrypoint.sh"
volumes:
- .:/alcor/
extra_hosts:
- "dockerhost:$DOCKERHOST" # for debugging
command:
# Here I omitted many options and arguments
- simulate
environment:
- CASSANDRA_RPC_ADDRESS=cassandra
- CASSANDRA_RPC_PORT=9042
cassandra:
image: cassandra:latest
volumes:
- cassandra-data:/var/lib/cassandra
volumes:
cassandra-data:
Edit: I tried again. I cleaned everything, rebooted and rebuilt, and the 1st error returned.
First of all, in docker-compose.yml I had to change the line
- "dockerhost:$DOCKERHOST" # for debugging
to
- "dockerhost:172.17.0.1"
where 172.17.0.1 is something called bridge network gateway.
It didn't completely solve the problem with the second error. But what helped was running the following commands:
sudo ufw disable
sudo systemctl restart docker
where the first one disables the firewall.
It looked like it was a problem specific to Ubuntu, as I never had this problem on Linux Mint, and on Debian Wheezy it happened very rarely.