I am wondering how to use tqdm with a script that also prints other messages besides the ones dedicated to monitor progression.
For example, suppose a loop like this, where three messages are written to stdout for each loop cycle:
for i in $(seq 10);
do
# This is the message used to control progress
echo "progress"
# Other messages, invalidate progess bar
echo "b"
echo "c"
# Simulate work
sleep 0.5
done | tqdm --total 10 --null;
If executed, this script shows a correct progess bar until 10 messages are written to stdout, then it goes back to the minimal progress bar as if it was not given the total, something like this:
30it [00:04, 6.01it/s]
This is because the total was set to 10, but more than 10 messages (30, actually) are sent through the pipe.
What I ended up writing is something like this, with the help of some other posts in StackOverflow and Unix & Linux Stack Exchange:
# Based on https://unix.stackexchange.com/a/537435/434897
{
for i in $(seq 10);
do
# This is the message used to control progress. sent to fd 3, only captured by tqdm
echo "progress" >&3
# Other messages, would invalidate progress report if sent directly to tqdm
echo "b"
echo "c"
# Simulate work
sleep 0.1
done 3>&1 >&4 | tqdm --total 10 --null;
} 4>&1
In summary, in the loop I write the progress messages to file descriptor 3 (echo "progress" >&3), all the other messages are written to stdout. Then, for the loop, I redirect 3 to stdout (3>&1), so that I can pipe those messages to tqdm, and redirect stdout to fd 4 (>%4), to avoid other messages being sent to the pipe and to tqdm. Finally, I redirect fd 4 to 1 (4>&1) for the whole loop + tqdm compound, so that the output is actually written to stdout and printed to the terminal.
Is there a more straightforward way to accomplish this behavior?
You can use a process substitution instead:
for i in $(seq 10);
do
# This is the message used to control progress
echo "progress" >&3
# Other messages, invalidate progess bar
echo "b"
echo "c"
# Simulate work
sleep 0.5
done 3> >(tqdm --total 10 --null)