My question is simple, how can I redirect all outputs of a bash script to file and terminal and remove color characters from within the script itself?
I can't find an answer which fits all my needs.
So far I tried tee to output to file and terminal, combined with 2>&1 to get stederr and stdout, sed to remove color characters and all of this with exec to do it from within my script but it doesn't work, I only get colored logs into the terminal and nothing in file.
#!/usr/bin/env bash
exec 2>&1 | sed -r 's/\x1b\[[0-9;]*m//g' | tee script.logs
python somepython.py
python someotherpython.py
Here the python scripts produce outputs which are colored.
I want to log those to terminal (untouched) and in file (without the color). In reality there is a lot more going on in my bash script than those just two python scripts, that's why I want to globally redirect the output of my script bash and not just pipe after each python script.
Thus I used exec because I though it allowed to redirect all output produce by a script.
Thanks in advance for any advice and help,
PS: I dont want colored logs in file but I don't care in terminal if this is what needs to be done for logs to not be colored in file.
You may put all your calls in a curly-braced group and redirect the whole lot, e.g.:
#!/usr/bin/env bash
{
python somepython.py
python someotherpython.py
} 2>&1 | sed -r 's/\x1b\[[0-9;]*m//g' | tee script.logs
This way, all stdout and stderr outputs will be passed along the filter.
If you want to write the colors to the terminal and write the uncolored text to the file, you may apply the sed filter to the file written by tee, e.g. your script would look something like:
#!/usr/bin/env bash
{
python somepython.py
python someotherpython.py
} 2>&1 | tee >(sed -r 's/\x1b\[[0-9;]*m//g' > script.logs)
This uses process substitution which is a very powerful tool in bash, albeit a bit difficult at first.
Assuming you would like to read the contents as soon as possible, you may want to deactivate python’s block buffering. This can be done using the -u option:
#!/usr/bin/env bash
{
python -u somepython.py
python -u someotherpython.py
} 2>&1 | tee >(sed -r 's/\x1b\[[0-9;]*m//g' > script.logs)
Not all special characters are based on the Control Sequence Initiator ESC [.
If a text is colored in green with tput setaf 1, it will use the control sequence ESC [31m, but if the color is reset with tput sgr0, the control sequence may be ESC (B ESC [m (please note the ESC (B sequence). So if you filter only the ESC [ sequences, you may still have control sequence waste in your log file.
Things get even worse if the program uses other types of control characters such as cursor commands.
For those reasons, the best way to avoid problems is to simply avoid writing control sequences from your python scripts. Most builtin programs are protected against that by checking if the output is a terminal before choosing to display colors or not. When the output is not a terminal it assumes that colors may cause issues.
With that said I don’t know if you have control over the python scripts (or other calls you might have), but if you do you might want to test if the output is a terminal. In Bash you check this way:
if [ -t 1 ] # does stdout end up on a terminal?
then
# Display fancy colors
else
# Minimalist display
fi
In Python it would be:
if sys.stdout.isatty(): # does stdout end up on a terminal?
# Display fancy colors
else:
# Minimalist display