PyTorch: Logging during model.fit()

Question

I'm using pytorch/fastai for training models. Since I'm working with remote machines, I am running the scripts using nohup python $1 >$2 2>&1 & with redirection to logging file like "log123.txt".

My problem is that during the model.fit() phase with scheduler, I can't see the progress in the file after each epoch like in console and the results are written to my logging file after the model.fit() finishes training. It works fine when I watch the process in console.

Is there any workaround for this?

Answer 1

This may be due to file buffering. You can disable it for python process like this:

1. Run your script with -u options. Example:

python -u my_script.py > log.txt

2. Use PYTHONUNBUFFERED:

PYTHONUNBUFFERED=1 python my_script.py > log.txt

or

export PYTHONUNBUFFERED=1
python my_script.py > log.txt