Search code examples
bashshellmove

Bash script to move folders based on filesize changes?

I have some automated downloads in a proprietary linux distro. They go to a temp scratch disk. I want to move them when they're finished to the main RAID array. The best way I can see to do this is to check the folders on the disk to see if the contents have changed in the last minute. If not then its probably finished downloading and then move it.

Assuming there could be hundreds of folders or just one in this location and its all going to the same place. Whats the best way to write this?

I can get a list of folder sizes with 

du -h directory/name

The folders can contain multiple files anywhere from 1.5mb to 10GB

Temp Loc:  /volume2/4TBScratch/Processing
Dest Loc when complete: /volume1/S/00 Landing

EDIT: Using this: 

find  /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec mv "{}"  "/volume1/S/00 Landing" \;
find: `/volume2/4TBScratch/Processing/test': No such file or directory
4.3#

yet it DOES copy the relevant folders and all files. But the error worries me that something might go wrong in the future.... is it because there is multiple files and it's running the same move command for EACH file or folder in the root folder? But since it moves it all on the first iteration it cant find it on the next ones?

EDIT2: Using Rsync

4.3# find  /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;
skipping directory newtest
skipping directory erw

RESOLVED: EDIT3 Resolved with the help in the comments below. Final script looks like this:

find  /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync -a --remove-source-files "{}" "/volume1/S/00 Landing" \;
find /volume2/4TBScratch/Processing -depth -type d -empty -delete

rsync to move folders and files but leaves empty root dir the next command finds empty folders and removes them.

Thanks all!

Solution

  • You can use GNU find with options -size for detecting files/folders of certain size and use mv with the -exec option to move to destination directory. The syntax is

    find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec mv "{}"  "/volume1/S/00 Landing" \;

    Using rsync

    find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;

    The size with a - sign to indicate less than the mentioned size which in this case is 10GB. A note on each of the flags used

    • -type d -> For identifying only the folders from the source path.
    • -maxdepth 1 -> To look only on the current source directory and not being recursive.
    • -exec -> Execute command following it.

    Alternatively, if you want to find files that are last modified over a certain time(minutes), find has an option for -mmin which can be set to a value. E.g. -mmin -5 would return files modified five minutes ago.

    So suggest adding it to your requirement, for x as you need and see if the directories are listed, then you can add the -exec option for moving the directories

    find /volume2/4TBScratch/Processing -type d -maxdepth 1 -mmin -2 -size -10G

    Refer to the GNU documentation for finding files according to size on how this works.

    Note:- The double quotes("") are added to avoid Bash from splitting the names containing spaces.