exit shell script from subshell if files are not present on remote server or disk is full?

Question

I am using below script to copy files in parallel from remote server:

#!/bin/bash

primary=/bat/primary
secondary=/bat/secondary

do_copy() {
  el=$1
  primsec=$2
  (scp -C -o StrictHostKeyChecking=no \
   goldy@"$remote":"$pod1"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
  || \
  (scp -C -o StrictHostKeyChecking=no \
   goldy@"$remote":"$pod2"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
  || \
  (scp -C -o StrictHostKeyChecking=no \
   goldy@"$remote":"$pod3"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)\
  || \
  (scp -C -o StrictHostKeyChecking=no \
   goldy@"$remote":"$pod4"/abc_187_"$el"_111_8.data "$primsec"/. > /dev/null 2>&1)
}
export -f do_copy

parallel -j "$5" do_copy {} $primary ::: ${pri[@]} &
parallel -j "$5" do_copy {} $secondary ::: ${snd[@]} &
wait
echo "all files copied"

On the remote server I have four folders represented by $pod1, $pod2, $pod3, $pod4. If files are not there in $pod1, then it should be there in $pod2, or it will be there in $pod3 or it will be there in $pod4 folder.

Now what I want to do is:

• If files are not there in any of those four folders then I want to exit from shell script with nonzero status code immediately.
• If for whatever reason let's say disk is full and file were not copied fully then I want to exit as well with nonzero status code. Each file size is around 15GB and sometimes because of disk full, I see only 3GB copied out of 15GB and my shell script don't report any issues and passes successfully.

I tried adding exit command next to last scp command but it just exits from the subshell not from whole script. Is there any way to do these things?

Answer 1

I will assume that the file exists on only one pod, so trying all pods is not a problem.

#!/bin/bash

primary=/bat/primary
secondary=/bat/secondary

# Activate parset and env_parallel if not already done
. `which env_parallel.bash`

# Make array with $primary as many times as elements in $pri
parset prim -N0 echo $primary ::: ${pri[@]} 
# Make array with $secondary as many times as elements in $pri
parset seco -N0 echo $secondary ::: ${sec[@]}
export remote

do_copy() {
  el=$1
  primsec=$2
  pod=$3
  scp -C -o StrictHostKeyChecking=no \
   goldy@"$remote":"$pod"/abc_187_"$el"_111_8.data "$primsec"/
}
export -f do_copy
copy_one_file() {
  parallel do_copy $1 $2 ::: pod{1..4}
  if [ $? == 4 ] ; then
    return 1
  else
    return 0
  fi
}
export -f copy_one_file
parallel --halt now,fail=1 copy_one_file :::  ${pri[@]} ${sec[@]} :::+ ${prim[@]} ${seco[@]}

parset makes an array with $primary as long as $pri.

:::+ links the elements from $pri to the elements in $prim.

You will need parallel version > 20170422 for parset.