What is the right way to increase the hard and soft ulimits for a singularity-container image?

Question

The task I want to complete: I need to run a python package inside of a singularity-container that is asking to open at least some 9704 files. This is the first I have heard of it and searching around this has something to do with a system’s ulimit.

What I currently have is the following def file.

I am setting the * hard nofile flag and the * soft nofile flag to 15 thousand. The sed line does edit the conf file but within the singularity shell my ulimit is still the default 1024.

Bootstrap: docker
From: fedora

%post
    dnf -y update
    dnf -y install nano pip wget libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver
    wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
    /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
    conda config --file /.condarc --add channels defaults
    conda config --file /.condarc --add channels conda-forge
    conda update conda
    sed -i '2s/#/\n* hard nofile 15000\n* soft nofile 15000\n\n#/g' /etc/security/limits.conf
    bash

%runscript
    python /Users/lamsal/count_of_monte_cristo/orthofinder_run/OrthoFinder_source/orthofinder.py -f /Users/lamsal/count_of_monte_cristo/orthofinder_run/concatanated_FAs/

I am following the “official” instuctions to change the ulimits for a RHEL based system from IBM’s webpage here: https://www.ibm.com/docs/en/rational-clearcase/9.0.2?topic=servers-increasing-number-file-handles-linux-workstations

Is the sed line not the right way to change ulimits for a singularity image?

Answer 1

Short answer:

Change the value on the host OS.

Long answer:

In this instance, running a singularity container is best thought of as any other binary you're executing in your host OS. It creates its own separate environment, but otherwise it follows the rules and restrictions of the user running it. Here, the ulimit is taken from the host kernel and completely ignores any configs that may exist in the container itself.

Compare the output from the following:

# check the ulimit on the host
ulimit -n

# check the ulimit in the singularity container
singularity exec -e image.sif ulimit -n

# docker only cares about container config settings
docker run --rm fedora:latest ulimit -n

# change your local ulimit
ulimit -n 4096

# verify it has changed
ulimit -n

# singularity has changed
singularity exec -e image.sif ulimit -n

# ... but docker hasn't
docker run --rm fedora:latest ulimit -n

To have a persistent fix, you'll need to modify the setting on your host OS. Assuming you're on MacOS this answer should take care of that.

If you don't have root privs or you're only using this intermittently you can run ulimit by before running singularity. Alternatively, you could use a wrapper script to run the image and set it in there.