Logging stderr and stdout to log file and handling errors in bash script

Question

RESOLUTION:

See Send stderr/stdout messages to function and trap exit signal

Edit I see that I have not been precise enough in my original post, sorry about that! So now I will incldue code example of my bash script with refined questions:

My four questions:

1. How can I send stdout to file and stderr to log() function in mysql command in STEP 2
2. How can I send stdout and stderr to log() function in gzip STEP 2 and rsync command STEP 3?
3. How to read stdout and stderr in log() function in question 1 and 2 above?
4. How can I still trap all errors in the onexit() function?

In generel: I want stdout and stderr to go to log() function so the messages can be put to specific log file according to specific format, but in some cases, like for mysqldump command, stdout need to go to file. Also, error occurs and is sent to stderr, I want to end up in onexit() function after the log statement is finished.

#!/bin/bash -Eu
# -E: ERR trap is inherited by shell functions.
# -u: Treat unset variables as an error when substituting.

# Example script for handling bash errors.  Exit on error.  Trap exit.
# This script is supposed to run in a subshell.
# See also: http://fvue.nl/wiki/Bash:_Error_handling

#  Trap non-normal exit signals: 2/INT, 3/QUIT, 15/TERM,
trap onexit 1 2 3 15 ERR

# ****** VARIABLES STARTS HERE ***********
BACKUP_ROOT_DIR=/var/app/backup
BACKUP_DIR_DATE=${BACKUP_ROOT_DIR}/`date "+%Y-%m-%d"`
EMAIL_FROM="foo@bar.com"
EMAIL_RECIPIENTS="bar@foo.com"
...
# ****** VARIABLES STARTS HERE ***********

# ****** FUNCTIONS STARTS HERE ***********
# Function that checks if all folders exists and create new ones as required
function checkFolders()
{
    if [ ! -d "${BACKUP_ROOT_DIR}" ] ; then
        log "ERROR" "Backup directory doesn't exist"
        exit
    else
        log "INFO" "All folders exists"
    fi

    if [ ! -d "${BACKUP_DIR_DATE}" ] ; then
        mkdir ${BACKUP_DIR_DATE} -v
        log "INFO" "Created new backup directory"
    else
        log "WARN" "Backup directory already exists"
    fi
}

# Function executed when exiting the script, either because of an error or successfully run
function onexit() {
    local exit_status=${1:-$?}

    # Send email notification with the status
    echo "Backup finished at `date` with status ${exit_status_text} | mail -s "${exit_status_text} - backup" -S from="${EMAIL_FROM}" ${EMAIL_RECIPIENTS}"
    log "INFO" "Email notification sent with execution status ${exit_status_text}"
   
    # Print script duration to the console
    ELAPSED_TIME=$((${SECONDS} - ${START_TIME}))
    log "INFO" "Backup finished" "startDate=\"${START_DATE}\", endDate=\"`date`\", duration=\"$((${ELAPSED_TIME}/60)) min $((${ELAPSED_TIME}%60)) sec\""
    
    exit ${exit_status}
}

# Logs to custom log file according to preferred log format for Splunk
# Input:
#   1. severity (INFO,WARN,DEBUG,ERROR)
#   2. the message
#   3. additional fields
#
function log() {
    local print_msg="`date +"%FT%T.%N%Z"` severity=\"${1}\",message=\"${2}\",transactionID=\"${TRANS_ID}\",source=\"${SCRIPT_NAME}\",environment=\"${ENV}\",application=\"${APP}\""

    # check if additional fields set in the 3. parameter
    if [ $# -eq 3 ] ; then
        print_msg="${print_msg}, ${3}"
    fi

    echo ${print_msg} >> ${LOG_FILE}
}
# ****** FUNCTIONS ENDS HERE ***********

# ****** SCRIPT STARTS HERE ***********
log "INFO" "Backup of ${APP} in ${ENV} starting"

# STEP 1 - validate
log "INFO" "1/3 Checking folder paths"
checkFolders

# STEP 2 - mysql dump
log "INFO" "2/3 Dumping ${APP} database"
mysqldump --single-transaction ${DB_NAME} > ${BACKUP_DIR_DATE}/${SQL_BACKUP_FILE}
gzip -f ${BACKUP_DIR_DATE}/${SQL_BACKUP_FILE}
log "INFO" "Mysql dump finished."

# STEP 3 - transfer
# Files are only transferred if all commands has been running successfully. Transfer is done with use of rsync
log "INFO" "3/3 Transferring backup file"
rsync -r -av ${BACKUP_ROOT_DIR}/ ${BACKUP_TRANSFER_USER}@${BACKUP_TRANSFER_DEST}

# ****** SCRIPT ENDS HERE ***********
onexit

Thanks!

Answer 1

Let's suppose that your log function looks like this (it just echos the first argument):

log() { echo "$1"; }

To save the stdout of mysqldump to some file and call your log() function for every line in stderr, do this:

mysqldump 2>&1 >/your/sql_dump_file.dat | while IFS= read -r line; do log "$line"; done

If you wanted to use xargs, you could do it this way. However, you'd be starting a new shell every time.

export -f log
mysqldump 2>&1 >/your/sql_dump_file.dat | xargs -L1 bash -i -c 'log $@' _