Search code examples
bashloopsforeach

Multiple looping with config file in bash

I want to copy latest file based on filename in (filename_config) & in each directory based on config file (dir_config) to backup directory.

In one directory could have > 1 filename to be copy.

dir_config:

/XXX/DIR1
/XXX/DIR2
/XXX/DIRB
/XXX/DIRY7

filename_config:

ABCD
123AB
E142

However my bash script it only copy 1 filename to the backup directory even though in the dir_config having multiple filename to be copy.

Example Output:

[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR1/12-ABCD_1.0_20231103082747609.zip to /srv/backup_data/XXX/DIR1
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR2/E142_1.0_20231103082747609.tar to /srv/backup_data/XXX/DIR2
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIRB/17-ABCD_1.0_20231103082747609.tar to /srv/backup_data/XXX/DIRB
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIRY7/XY-123AB_1.0_20231103082747609.csv to /srv/backup_data/XXX/DIRY7

This my script:

for i in $(<$dir_config)
do
    cd $Source_Dir
    
    for y in $(<$filename_config)
    do
        latest_file=$(LC_ALL=C ls $i *$y* -cr  | tail -1)
    done
    
    cp "$Source_Dir$i/$latest_file" $Backup_Dir$i
    WriteLog "Copy latest file from $Source_Dir$i/$latest_file to $Backup_Dir$i "
    
done

I'm expecting as below:

Output:

[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR1/12-ABCD_1.0_20231103082747609.zip to /srv/backup_data/XXX/DIR1
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR1/E142_1.0_20231103082747601.zip to /srv/backup_data/XXX/DIR1
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR1/ZA-123AB_1.0_20231103082747601.zip to /srv/backup_data/XXX/DIR1
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR2/E142_1.0_20231103082747607.tar to /srv/backup_data/XXX/DIR2
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIR2/12-ABCD_1.0_20231103082747602.tar to /srv/backup_data/XXX/DIR2
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIRB/17-ABCD_1.0_20231103082747604.tar to /srv/backup_data/XXX/DIRB
[Wed Jul 10 13:55:09 UTC 2024]   Copy latest file from /srv/XXX/DIRY7/XY-123AB_1.0_20231103082747605.csv to /srv/backup_data/XXX/DIRY7
Solution

  • Making a few assumptions - given that

    • $Source_Dir, $filename_config, and $dir_config are being set with relevant full or relative paths,
    • /XXX/ is always the same $Source_Dir,
    • dir_config does NOT have full pathnames, but subfolders of $Source_Dir,
    • each $Source_Dir might have several matches from filename_config
    • WriteLog is a function or something that you defined elsewhere

    then refactor a bit.

    while read -r d 
do while read -r f 
   do  read -r x x x latest < <( 
         stat -c "%z %n" "$Source_Dir/$d"/*"$f"* | sort -nr ) || continue
       cp -put "$Backup_Dir/$d/" "$Source_Dir/$d/$latest"
       WriteLog "$Source_Dir/$d/$latest => $Backup_Dir/$d/"
   done < "$filename_config"
done < "$dir_config"

    This doesn't care where it's run and doesn't need to change directories, assuming you are setting $Source_Dir, $filename_config, and $dir_config with relevant full or relative paths.

    It doesn't try to parse ls, quotes all the vars, preserves file attributes, doesn't bother copying files that aren't newer than one with the same name already in the $Backup_Dir's matching subfolder, and doesn't try to copy a file if it didn't exist in the $Source_Dir's subdirectory, though it will report the error if one is missing.

    Easy to do if you'd rather check that and skip the error messages, but please don't just throw the errors to the bitbucket...

    Note that I put the copy inside the loop that reads $filename_config. If a single directory ever had files that matched more than one of your listed patterns you were only getting the last one - Look at this example I set up.

    $: ls -l
total 1
-rw-r--r-- 1 P2759474 1049089 8 Jul 11 09:49 filename_config
-rw-r--r-- 1 P2759474 1049089 0 Jul 11 09:48 foobar1
-rw-r--r-- 1 P2759474 1049089 0 Jul 11 09:48 foobar2
-rw-r--r-- 1 P2759474 1049089 0 Jul 11 09:48 other1
-rw-r--r-- 1 P2759474 1049089 0 Jul 11 09:48 other2

$: echo "$filename_config:"; cat $filename_config
filename_config:
foo
her

$: for y in $(<$filename_config); do  latest_file=$(LC_ALL=C ls $i *$y* -cr  | tail -1); echo "$latest_file"; done
foobar2
other2

$: echo "cp $latest_file \$somewhere"
cp other2 $somewhere

    This is what your code is doing. It should back up foobar2, but the loop is assigning it, then moving on to the next file and overwriting it before you do the copy.

    You could do it that way with an array if you want -

    $: for y in $(<$filename_config); do latest_file+=( $(LC_ALL=C ls $i *$y* -cr  | tail -1) ); done
$: echo cp "${latest_file[@]}" \$somewhere
cp foobar2 other2 $somewhere

    Not an issue if a given directory can only have one match, but then if that's the case why do you need two config files? If one directory always has only one match, you could set up one file with both pattern and directory and do the whole thing in one loop.