Search code examples
stdoutslurm

How to change how frequently SLURM updates the output file (stdout)?

I am using SLURM to dispatch jobs on a supercomputer. I have set the --output=log.out option to place the content from a job's stdout into a file (log.out). I'm finding that the file is updated every 30-60 minutes, making it difficult for me to check on the status of my jobs.

Any idea why it takes so long to update this file? Is there a way to change settings so that this file is updated more frequently?

Using SLURM 14.03.4-2

Solution

  • This may be related to buffering.

    Have you tried disabling output buffering as suggested in here? I would recommend the stdbuf option:

    stdbuf -o0 -e0 command

    But can't be sure without more information, as I've never experienced a behavior like that. Which filesystem are you using?

    Also if you are using srun to run your commands you can use the --unbuffered option which disables the output buffering.