I am using Azure Batch to execute a process which has different steps, the process is programmed in Python. I want to keep a log of the steps my process is in in the stdout.txt file in Azure Batch.
So far, I am printing the step I am in. With this, once the batch task is completed, the prints will appear in the stdout.txt. However, I couldn't find the way to update the stdout.txt file in real time.
For instance, imagine I have the following code snippet:
print('Step 1')
# Step 1 code
print('Step 2')
# Step 2 code
print('Step 3')
# Step 3 code
Once my whole task is executed, the stdout.txt will look like this:
Step 1
Step 2
Step 3
However, during the process, if I try to access to the file, it will be empty.
I am wondering if there is any way to get these prints in real time, not having to wait until the task is completed to have something written down, i.e. while the task is still in the Step 1, the only line in my stdout.txt would be:
Step 1
And the file would be updated when a new step starts (and therefore the print). Any idea or hint?
This is most likely a side effect of your program buffering output. If you need to see data in the stdout stream immediately, you will likely need to flush. You can attempt to flush stdout in Python or start Python in unbuffered mode.