Search code examples
linuxunixmultiprocessingcgroups

How to increase number of child proceses


cat /sys/fs/cgroup/pids/parent/pids.max   =  "max"  

I created it following https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/pids.html

Consider this Python Code demonstrating the problem:

from os import fork, getpid
from time import sleep


i=0

print( "pid = %d " % getpid())

with open("/proc/%d/limits" % getpid(), "r") as f:
    print(f.read())


try:
    while fork():
        i+=1

except BaseException as e:
    print(i)
    print(e)
    sleep(10)
    print("done")

exit(1)

My output:

pid = 18091 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             999999               999999               processes 
Max open files            1024                 1048576              files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31412                31412                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

10227
[Errno 11] Resource temporarily unavailable
done

Solution

  • In Linux you have the pid_max limit:

    $ cat /proc/sys/kernel/pid_max
    32768
    

    However, if your Linux is running on systemd you might hit user-slice limits:

    for root

    $ cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max
    

    for currently logged in user:

    $ cat /sys/fs/cgroup/pids/user.slice/user-$(id -u).slice/pids.max 
    10813
    

    The systemd equivalent would be:

    $ systemd-analyze dump | sed -n "/-> Unit user-$(id -u).slice:/,/-> Unit /p"| grep -e "TasksMax="
            TasksMax=10813
    

    According to man logind.conf

      UserTasksMax=
          Sets the maximum number of OS tasks each user may run concurrently. > This controls the TasksMax= setting of the per-user slice unit, see  systemd.resource-control(5) for details. If assigned the special value
          "infinity", no tasks limit is applied. Defaults to 33%, which equals 10813 with the kernel's defaults on the host, but might be smaller in OS containers.
    

    This limit is defined in /etc/systemd/logind.conf, even though it might be commented out, 33% out of 32768.

    #UserTasksMax=33%
    

    When you modify the UserTasksMax limit or increase sysctl kernel.pid_max you'll have to restart systemd-logind service:

    service systemd-logind restart
    

    On 64-bit system you should be able to increase the value up to 2^22, i.e.: 4194304

    sysctl kernel.pid_max=4194304