I am writing a script to build a CSV file to import to our LMS. One of the script's function does an ldapsearch for the user's guid. The command looks as such:
ldapsearch -x -h ldap.example.com -b ou=person,dc=example,dc=com user=username
On its own the command works fine. However, in the function as part of while loop it fails. When I set xtrace in bash I see the command is outputted with the following:
ldapsearch -x -h ldap.example.com -b ou=person,dc=example,dc=com $'user=username\r'
After hours of searching it seems that the $' \r' are known as ANSI C quotes, which I do not have a lot of experience with. The question is, why is the argument getting wrapped with these quotes in the first place? and second, how to I escape them?
Now, here is the kicker, this script has 3 functions which are run depending on the script's argument. User function, Enrollment function and Course function (no complete yet). The issue is only happening in the Enrollment function, however in User function, the same loop is working 100% no issues, and the code is line per line identical, so what gives?
Any help with this would be greatly appreciated, even more if it is before I pull whats left of my hair out.
Below are the variables and affected function code:
The affected line is:
found=$(${ldap_cmd}=${course_instructor} | grep numEntries: | awk -F: '{print $NF}' | sed -e 's/^[ \t]*//')
This line tests to see if the user exists in LDAP or not.
#!/bin/bash
# Set Global variables here
export PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
export ldap_host="ldap.example.com"
export ldap_base="ou=person,dc=example,dc=com"
export ldap_cmd="ldapsearch -x -h ${ldap_host} -b ${ldap_base} user"
export script_name=$(basename "$0"| cut -d. -f1)
export script_pid=$(pgrep "${script_name}")
export admin_id=$(whoami)
export admin_mail=$(${ldap_cmd}="${admin_id}" | grep mail: | awk -F: '{print $NF}' | tail -1 | sed -e 's/^[ \t]*//')
export admin_name=$(${ldap_cmd}="${admin_id}" | grep givenName: | awk -F: '{print $NF}' | tail -1 | sed -e 's/^[ \t]*//')
export admin_fname=$(${ldap_cmd}="${admin_id}" | grep cn: | awk -F: '{print $NF}' | tail -1 | sed -e 's/^[ \t]*//')
function enrollment_feed () {
# Fountion specific variables
# enrollment_feed ${input_option} ${term_option} ${input_file}
export input_count=$(wc -l "${input_file}" | awk '{print $1}')
export enrollment_file="$(pwd)/erollments.csv"
export error_file="$(pwd)/error.csv"
# Create the output files enrollment and error
echo "course_id,root_account,user_id,role,role_id,section_id,status,associated_user_id,limit_section_privileges" > "${enrollment_file}"
echo "error" > "${error_file}"
# main script while loop
clear
printf "\n========================================="
printf "\n= Canvas SIS Import Builder Tool ="
printf "\n= Enrollment Feed ="
printf "\n========================================="
printf "\n${inf_msg} Preparing to run script ${orange}${script_name}${no_color}"
sleep 0.5
printf "\n${inf_msg} The script process ID is ${orange}${script_pid}${no_color}"
sleep 0.5
printf "\n${inf_msg} The script in run by: ${orange}${admin_fname}${no_color}"
sleep 0.5
printf "\n${inf_msg} The script will start running now"
printf "\n========================================="
printf "\n= Started $(date) ="
printf "\n========================================="
printf "\n"
sleep 0.5
while IFS="," read -r course_title course_number course_section course_instructor; do
if ! [[ "${course_number}" =~ ^[0-9]+$ ]]; then
printf "\r${war_msg} header or blank line detected"
tput el
sleep 0.5
printf "\r${war_msg} Skipping line.."
tput el
sleep 0.5
printf "\r"
tput el
sleep 0.5
printf "\r${bang_mark} Line Skipped.\n"
else
printf "\r${inf_msg} Start processing ${orange}${input_term}-${course_number}-${course_section}${no_color}..."
tput el
sleep 0.5
printf "\r${inf_msg} Searching directory services for the user record matching AndrewID ${orange}${course_instructor}${no_color}"
tput el
sleep 0.5
found=$(${ldap_cmd}=${course_instructor} | grep numEntries: | awk -F: '{print $NF}' | sed -e 's/^[ \t]*//')
if [[ "${found}" = "1" ]]; then
printf "\r${inf_msg} A user record was found in directory services matching AndrewID ${orange}${course_instructor}${no_color}"
tput el
sleep 0.5
printf "\r${inf_msg} Collecting all required fields from directory services matching AndrewID ${orange}${course_instructor}${no_color}"
tput el
sleep 0.5
printf "\r${inf_msg} Constructing information for ${orange}${input_term}-${course_number}-${course_section}${no_color}"
tput el
sleep 0.5
course_id="${input_term}-${course_number}-${course_section}"
root_account="canvas.cmu.edu"
user_id=$(${ldap_cmd}="${course_instructor}" | grep guid: | awk -F: '{print $NF}' | sed -e 's/^[ \t]*//')
role="teacher"
role_id="4"
section_id=""
status="active"
associated_user_id=""
limit_section_privileges=""
printf "\r${inf_msg} Exporting all required fields for course ${orange}${input_term}-${course_number}-${course_section}${no_color} to the enrollments csv file"
tput el
sleep 0.5
echo ${course_id},${root_account},${user_id},${role},${role_id},${section_id},${status},${associated_user_id},${limit_section_privileges} >> "${enrollment_file}"
printf "\r${inf_msg} All required fields for course ${orange}${input_term}-${course_number}-${course_section}${no_color} have now been exported to the enrollment csv file"
tput el
sleep 0.5
printf "\r"
tput el
sleep 0.5
printf "\r${check_mark} Couse/section ${orange}${input_term}-${course_number}-${course_section}${no_color} Done.\n"
else
printf "\r${war_msg} The was no user record matching AndrewID ${blue}${course_instructor}${no_color} in directory services"
tput el
sleep 0.5
printf "\r${war_msg} Roprting this in error csv"
tput el
sleep 0.5
printf "\r"
tput el
sleep 0.5
printf "\r${cross_mark} Instructor for course ${red}${input_term}-${course_number}-${course_section}${no_color} not found.\n"
echo "${input_term}-${course_number}-${course_section}" >> "${error_file}"
fi
fi
done < "${input_file}"
printf "\n========================================="
printf "\n= Finished $(date) ="
printf "\n========================================="
printf "\n"
sleep 0.5
# send files in email to admin
printf "\n${inf_msg} Sending email with csv files attached to ${orange}${admin_mail}${no_color}"
sleep 0.5
msg_body="Dear ${admin_name},\n\nPlease find attached the following two files:\n\n - enrollments.csv: (This file contains all the Instructor information needed for and enrollments SIS import in Canvas)\n - error.csv: (This file contains course id for course with no valid instrcutor id)\n\nPlease use these files responsibly and in line with CMU privcy guidelines.\n\nBest regards,\nCanvas Admin\n\n\nThe script "${script_name}" was run by ${admin_id} from ${computer_name} at $(date +"%Y-%m-%d %T%Z")"
printf "${msg_body}" | mailx -s "Enrollments list as of $(date)" -a "${user_file}" -a "${email_file}" -a "${error_file}" "${admin_mail}"
printf "\n${inf_msg} Email has been sent, please check your mailbox"
sleep 0.5
printf "\n${inf_msg} The script has now finished processing all ${input_count} records"
sleep 0.5
printf "\n${inf_msg} Deleting the temp csv files"
rm -rf "${enrollment_file}"
sleep 0.5
rm -rf "${error_file}"
sleep 0.5
printf "\n${inf_msg} The script ran for ${orange}$(ps -p "${script_pid}" -o etime | tail -1 | awk '{print $1}')${no_color}"
sleep 0.5
printf "\n${inf_msg} The script is now exiting"
sleep 0.5
printf "\n"
printf "\n========================================="
printf "\n= Good Bye! ="
printf "\n========================================="
printf "\n"
exit 0
}
The problem apparently comes from this line
while IFS="," read -r course_title course_number course_section course_instructor ; do
where you can see that course_instructor is read from the end of the line - and when read from a 'wrong' terminal which sends \r as part of the newline, the \r ends up in the variable. The $'something\r' notation is only a way how to include special characters in a bash command line.
(guessing) The other functions which work well for you probably don't use the last variable (which includes \r) in a place where it fails immediately.
The quick fix is a) use better terminal for data entry, b) get rid of the \r by eg <<<"$var" tr -d '\r'.