LSF job submission - stdout & stderr redirection

I've submitted my job by the following command:

bsub -e error.log -o output.log ./myScript.sh

I have a question: why are the output and errors logs available only once the job ended?

Thanks

Solution

LSF doesn't steam the output back to the submission host. If the submission host and the execution host have a shared file system, and the JOB_SPOOL_DIR is in that shared file system (the spool directory is $HOME/.lsbatch by default) then you should see the stdout and stderr there. After the job finishes, the files there are copied back to the location specified by bsub.

Check bparams -a | grep JOB_SPOOL_DIR to see if the admin has changed the location of the spool dir. With or without the -o/-e options, while the job is running its stdout/err will be captured in the job's spool directory. When the job is finished, the stdout/stderr is copied to the filenames specified by bsub -o/-e. The location of the files in the spool dir is $JOB_SPOOL_DIR/<jobsubmittime>.<jobid>.out or $JOB_SPOOL_DIR/<jobsubmittime>.<jobid>.err

[user1@beta ~]$ cat log.sh 
LINE=1
while :
do
  echo "line $LINE"
  LINE=$((LINE+1))
  sleep 1
done

[user1@beta ~]$ bsub -o output.log -e error.log ./log.sh
Job <930> is submitted to default queue <normal>.
[user1@beta ~]$ tail -f .lsbatch/*.930.out
line 1
line 2
line 3
...