I am running a local docker-compose setup which has one Zookeeper container (latest is fetched) and one Marathon container (v1.1.0). Since last month whenever I up my containers my CPU usage goes to 100% after couple of minutes even after I keep it idle and not do any operations.
My setup - Ubuntu 17.10 with docker-ce 18.03.0 and docker-compose 1.21.0
Running top gives me this -
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7055 root 20 0 4527380 446644 45836 S 99.0 6.5 2:10.43 java
ps aux | grep java gives this -
harry 6928 0.1 0.6 3227008 47656 ? Ssl 16:00 0:02 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /zookeeper-3.4.11/bin/../build/classes:/zookeeper-3.4.11/bin/../build/lib/*.jar:/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg
root 7055 11.9 6.6 4527380 459896 ? Ssl 16:00 3:35 java -jar /marathon/bin/../target/marathon-assembly-1.1.1.jar --master local --zk zk://zookeeper:2181/marathon --event_subscriber http_callback
harry 8551 0.0 0.0 15376 1068 pts/4 S+ 16:30 0:00 grep --color=auto java
I have confirmed with my colleagues running OSX and the same setup isn't affecting them. So this has to do something with the marathon or zookeeper container doing something wrong on Ubuntu.
First of all, you should check if 100% is mainly due to your container execution or other programs in host. It's not very clear if your cpu load is caused by containers proccess or host process, so, I'll set out both possibilities.
If high cpu load is mainly caused by containers.
In this case, you should run docker containers limiting cpu load (quota) for host:
docker run --cpu-quota (int value) ...
Of course, it's also useful set a docker execution on a concrete core/cpu of your host. For example:
docker run --cpuset-cpus 0,1,5 ...
docker run --cpuset-cpus 0-4 ...
If you want to priorize docker execution in case a high load: It'll be useful launch docker container specifying how many cpus you want to reserve for your docker container. :
NCPUS=`cat /proc/cpuinfo | grep processor | wc -l`
docker run --cpus [float number between 0 and $NCPUS] ...
Note that 100% of all cpu cores is equivalent to execute docker run --cpus $NCPUS, but this is not possible obviously because your system needs some cpu usage. Furthermore, if you use equivalent option with swarm replicas, for instance, replicas will only start if there's enough cpu resources. See documentation: docker info