Search code examples
dockercgroups

What's other resource limitation of docker besides CPU & memory?


I'm running two docker container vm1 and vm2 with the same docker image. Both can run successful separately. But can not run at the same time. I've checked the CPU and memory. What's other resource limited the docker to run multiple containers?

systemctl status docker result when run 'vm1' only.

   Tasks: 148
   Memory: 357.9M

systemctl status docker result when run 'vm2' only.

   Tasks: 140
   Memory: 360.0M

My system still contains about 4GB free RAM, the CPUs are idle too. When I run vm1 then vm2, the vm2 will failed with some log like:

[17:55:50.504452] connect(172.17.0.2, 2889) failed: Operation now in progress 

And other log like

/etc/bashrc: fork: retry: Resource temporarily unavailable

systemctl status docker result when run 'vm1' then 'vm2'.

   Tasks: 244
   Memory: 372.2M

vm1 docker run command

  exec docker run --rm -it --name vm1
    -e OUT_IP="$MYIP" \
    -h vm1 \
    -v /MyLib/opt:/opt:ro \
    -v /home/myid:/home/guest \
    -v /sybase:/sybase \
    -v /sybaseDB:/sybaseDB \
       run-image $*

vm2 docker run command

  exec docker run --rm -it --name vm2
    -e OUT_IP="$MYIP" \
    -h vm2 \
    -v /MyLib/opt:/opt:ro \
    -v /home/myid:/home/guest \
    -v /sybase2:/sybase \
    -v /sybaseDB2:/sybaseDB2 \
       run-image $*

Some command result according to: fork: retry: Resource temporarily unavailable

# in host os
$ sysctl fs.file-nr
fs.file-nr = 4064 0 814022

# in docker container (vm2)
$ sudo su - guest
$ ulimit -Hn
1048576
$ sudo lsof -u guest 2>/dev/null | wc -l
230

The docker run user is 'guest', but I run program by 'ap' user account through sudo. I found there is different of 'ulimit -u' result inside the container, the run-image is based on centos:6

$ sudo su - guest
$ ulimit -u
unlimited
$ sudo su - ap
$ ulimit -u
1024

Solution

  • In my case, I found the result is caused by 'ap' user's default ulimit -u is only 1024. When run only vm1 or vm2, the 'ap' user's process/thread count is under 1024. When I run both vm1 and vm2, the total process count is larger than 1024.

    The solution is enlarge the default user nproc limitation for centos 6:

    sudo sed -i 's/1024/4096/' /etc/security/limits.d/90-nproc.conf