Search code examples
cronheredoclsf

script to log the completion of an LSF (bsub) job

I have a script called by cron to run an LSF job.

I would like to be told when that job is submitted and when it completes. The-Powers-That-Be have decided to disable email notifications. So I am writing this script to output the relevant information to a logfile.

It almost works. Here is some code:

crontab:

00 12 * * * my_script.sh my_job.bsub

my_job.bsub:

#!/bin/bash
#BSUB -q my_queue
#BSUB -W 06:00

echo "I am a long process"
sleep 60

my_script.sh:

#!/bin/sh

BSUB_SCRIPT=$1

# run bsub_script (and get the LSF job id while you're at it)...
JOB_ID=`bsub < $BSUB_SCRIPT | awk -F[\<,\>] '{print $2}'`

# log that job was submitted...                
echo "`date +%Y-%m%d %T` submitted '$BSUB_SCRIPT' [$JOB_ID]" >> $HOME/my_logfile.txt

# and log when job completes...
bsub -w "ended($JOB_ID)" << EOF
#!/bin/bash
#BSUB -q my_queue
#BSUB -W 00:30

echo "`date +%Y-%m-%d %T` completed '$BSUB_SCRIPT' [$JOB_ID]" >> $HOME/my_logfile.txt

EOF

(I found this answer helpful in figuring out how to submit a job that waits until an earlier one has completed.)

The issue is that the 2nd call to date, in the heredoc, gets evaluated immediately, so I wind up with a logfile that looks like this:

my_logfile.txt:

2018-01-30 13:15:14 submitted 'my_job.bsub' [1234567]
2018-01-30 13:15:14 completed 'my_job.bsub' [1234567]

Notice how the times are exactly the same.

How can I ensure that evaluation of the content of the heredoc is deferred until the LSF job runs?

Solution

  • The date command in the heredoc is being expanded before being passed to bsub. You need to quote the EOF in your heredoc expression or escape the date command. See the answer to this question: How does "cat << EOF" work in bash?

    In particular:

    The format of here-documents is:

          <<[-]word
              here-document
      delimiter

    ...

    If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion.

    So, for example when I run

    $ cat << EOF
> echo `date`
> EOF

    The output is

    echo Tue Jan 30 11:57:32 EST 2018

    Note that the date command is expanded, which is what's happening in your script. However, if I quote the delimiter in the heredoc:

    $ cat << "EOF"
> echo `date`
> EOF

    You get the unexpanded output you want:

    echo `date`

    Similarly, escaping date would preserve the other variables you want to expand:

    $ cat << EOF
> echo \$(date)
> EOF

    Output:

    echo $(date)