Search code examples
shellarchive

Move log files to archive folder by date, but not the last log file for each process


I have these files, [processName.DateTime.out]

/stev/log: ls -al

drwxrwxr-x    1 user.stev           user         4096 Feb  5 23:59 .
drwxrwxr-x    1 user.stev           user         4096 Feb  5 23:59 ..
-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 AAA01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 AAA01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 AAA01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  1 23:59 BBB01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  2 23:59 BBB01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  3 23:59 BBB01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  1 23:59 DDD01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  2 23:59 DDD01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  3 23:59 DDD01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  1 23:59 TEST01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  2 23:59 TEST01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  3 23:59 TEST01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  1 23:59 TEST02.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  2 23:59 TEST02.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Feb  3 23:59 TEST02.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 TEST03.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 TEST03.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 TEST03.202002030000.out

And my script moves some files to archive folder if it is modified before 7 days.

sfind . -type f -mtime +7 -name "/stev/log" | while read FNAME
    do
        echo " moving file: " ${FNAME}      >> /stev/archive/archiveLog.log
        PCMD="mv ${FNAME} /stev/archive}"
        echo "PCMD=${PCMD}"                 >> /stev/archive/archiveLog.log
        eval "${PCMD}"                      >> /stev/archive/archiveLog.log
        echo ""                             >> /stev/archive/archiveLog.log
    done

this script will list below files then move them to the archive folder [Current date: Feb 5]

-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 AAA01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 AAA01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 AAA01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 TEST03.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 TEST03.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 TEST03.202002030000.out

But the thing is, I have to remain the last files for each process.

For example) I can move belows files

-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 AAA01.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 AAA01.202002020000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 21 23:59 TEST03.202002010000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 22 23:59 TEST03.202002020000.out

but I should remain belows files (It doesn't matter how many days ago, for the last file for each process has to be remained.)

-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 AAA01.202002030000.out
-rw-rw-r--    1 user.stev           user         4096 Jan 23 23:59 TEST03.202002030000.out

Is there any good idea for this problems??


Solution

  • Think in bash as in "streams" - something goes in, something goes out.

    # find all files
    find /stev/log -type f -mtime +7 |
    # extract the part before dot
    cut -d. -f1 |
    # for each process
    # could be transformed into `xargs sh -c '...' --` for speed
    while IFS= read -r process; do
        # all files with this process
        find /stev/log -type f -mtime +7 -name "$process*" |
        # sort numerically second field behind dot
        sort -t. -n -k2 |
        # remove last file, because we don't want to move it
        # maybe you need to remove first line, then just sort -r, check it
        head -n -1
    done |
    # mv each file into /stev/archive
    xargs -d$'\n' -I{} mv -v {} "/stev/archive" >> /stev/archive/archiveLog.log
    

    Pass -t option to xargs to debug it.