LSF issue with large scale runs

Question

On our cluster, when I submit jobs requesting more than (including) 40 nodes or 640 cores, the $LSB_HOSTS gets empty and so the job stops. I use this variable to generate a nodelist file which I use with the mpirun command line as the following:

#BSUB -q cpu
#BSUB -J gromacs
#BSUB -o job.out
#BSUB -e job.err
#BSUB -n 640

#####################################################
#####################################################

INPUT=test184000atoms_verlet.tpr

echo ""
echo "----------------------- INTIALIZATIONS -----------------------------"
echo ""

source /lustre/utility/intel/composer_xe_2013.3.163/bin/compilervars.sh intel64
source /lustre/utility/intel/mkl/bin/intel64/mklvars_intel64.sh
source /lustre/utility/intel/impi/4.1.1.036/bin64/mpivars.sh

MPIRUN=/path/to/intel/impi/4.1.1.036/intel64/bin/mpirun

EXE=mdrun_mpi

if test ! -x `which $EXE` ; then
    echo
    echo "ERROR: `which $EXE` not existent or not executable"
    echo "Aborting"
    exit 1
fi

CURDIR=$PWD
cd $CURDIR
rm -f nodelist >& /dev/null
touch nodelist

for host in `echo $LSB_HOSTS`
do
echo $host >> nodelist
sleep 2
done

NP=`cat nodelist |wc -l`
NN=`cat nodelist |sort |uniq|tee nodes |wc -l`

echo
echo "Executable : `which $EXE`"
echo "Working directory is $CURDIR"
echo "Running on host `hostname`"
echo "Directory is `pwd`"
echo "This jobs runs on $NN nodes"
echo "This job has allocated $NP core(s)"
echo

ulimit -aH
echo
ls -al

echo ""
echo "----------------------- RUN -----------------------------"
echo ""

date '+RUN STARTED ON %m/%d/%y AT %H:%M:%S'

$MPIRUN -np $NN -machinefile nodes $EXE -v -deffnm  $INPUT >& $EXE.log

date '+RUN ENDED ON %m/%d/%y AT %H:%M:%S'

echo ""
echo "----------------------- DONE ----------------------------"
echo ""

ls -al

Any hints here?

Can you see something wrong with this script?

Thanks,

Éric.

Answer 1

I finally got a solution.

The point is that for some reason the variable LSB_HOSTS is sometime not set. Fortunately, there is another one: LSB_MCPU_HOSTS

For people who are interested, here is how I use it:

CURDIR=$PWD
cd $CURDIR
rm -f nodelist nodes n >& /dev/null
touch nodelist
touch nodes
NP=0

for host in `echo $LSB_MCPU_HOSTS | sed -e 's/ /:/g'| sed 's/:n/\nn/g'`
do
echo $host >> nodelist
echo $host | cut -d ":" -f1 >> nodes
nn=`echo $host | cut -d ":" -f2`
NP=`echo $NP+$nn | bc`
done

NN=`cat nodelist | wc -l`

echo
echo "Executable : `which $EXE`"
echo "Working directory is $CURDIR"
echo "Running on host `hostname`"
echo "Directory is `pwd`"
echo "This jobs runs on $NN nodes"
echo "This job has allocated $NP core(s)"
echo

Thank you for your help.

Éric.