Search code examples
perllsf

How to automate the LSF waiting based on job name in perl


I have a perl code where I am submitting few jobs at once in parallel via LSF bsub command and once all these jobs finish want to submit a final job.

For example I have these three bsub commands where first two bsub commands submits jobs t1 and t2 and third command checks whether t1 and t2 are finished or not and wait on them with -w argument.

   system(" bsub -E "xyz" -n 1 -q queueType -J t1 sleep 30")
    system("bsub -E "xyz" -n 1 -q queueType -J t2 sleep 30")
    system("bsub -E "xyz" -n 1 -q queueType -J t3 -w "done(t1)&&done(t2)" sleep 30")

So for automating -w argument I have this

    my $count=2;
    my $i; 
   system(" bsub -E "xyz" -n 1 -q queueType -J t3 \"foreach my $i (0..$count) {print " done(t_$i)&&";}\" sleep 30 ")

I get this error:

sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `bsub -E "/pkg/ice/sysadmin/bin/linux-pre-exec" -n 1 -q short -J t3 -w "foreach  (0..7) {print \"done(t)&&\";}" sleep 30'

EDIT: Yes I am using system command to submit these jobs from perl


Solution

  • If you want to generate the done(...)&&done(...) string dynamically, you can use

    my $count = 7;
    my $done_all = join '&&', map "done(t$_)", 1 .. $count;
    

    That is, for each number in the range 1 .. 7, produce a string "done(t$_)", which gives a list "done(t1)", "done(t2)", ... "done(t7)". The elements of this list are then join'd together with a separator of &&, producing "done(t1)&&done(t2)&&...&&done(t7)".

    To run an external command, you can use system. Passing a list to system avoids going through the shell, which avoids all kinds of nasty quoting issues:

    system('bsub', '-E', 'xyz', '-n', '1', '-q', 'queueType', '-J', 't3', '-w', $done_all, 'sleep', '30');
    # or alternatively:
    system(qw(bsub -E xyz -n 1 -q queueType -J t3 -w), $done_all, qw(sleep 30));
    

    Your code tries to pass Perl code to bsub, but that won't work. You have to generate the command string beforehand and pass the result to bsub.