Search code examples
bashrsyncpid

Bash Script with Handling PID's from Remote Server

I have boxes pulling video content down via RSYNC. +- 700 boxes, where I am currently testing with four boxes.

Each box has a bash script on it to pull the content down via a reverse RSYNC from a centralized server.

I would like to manage these connections to the server with the following:

  1. Connections must be limited to 50 connections max for RSYNC for this specfic task only (Not RSYNC as a whole, as other tasks depend on it).

  2. If the connection limit in 1. is reached, then it should kill off the earliest PID created.

The data below is presented as the following on the server, where the videos sit.

root@TESTSRV01:~# ps aux | grep rsync
1029     13357  0.0  0.0   4308   604 ?        Ss   11:46   0:00 sh -c rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/
1029     13358  0.1  0.0  30852  1444 ?        S    11:46   0:00 rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/
1029     13382  0.0  0.0   4308   604 ?        Ss   11:47   0:00 sh -c rsync --server --sender -vre.iLsf --bwlimit=100 --append --append . /home/test/Videos/
1029     13383  0.1  0.0  39432  1800 ?        S    11:47   0:00 rsync --server --sender -vre.iLsf --bwlimit=100 --append --append . /home/test/Videos/
1029     13400  0.0  0.0   4308   604 ?        Ss   11:47   0:00 sh -c rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/
1029     13401  0.1  0.0  39432  1800 ?        S    11:47   0:00 rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/
1029     13451  0.0  0.0   4308   608 ?        Ss   11:48   0:00 sh -c rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/
1029     13452  0.0  0.0  71128  2248 ?        S    11:48   0:00 rsync --server --sender -vre.iLs --bwlimit=100 --append --append . /home/test/Videos/

From the script on the box, I have the following so far:

ps -u test | grep rsync | awk '{ print $1 }'

This returns the following: (Which are the PID's)

13358
13383
13401
13452

Keeping in mind, point 1. and 2. how would I achieve this. Thanks!

Solution

  • You could do this. Extract all pids sorted by start_time and put them in array.

    declare -a pids
pids=($(ps -u test --sort=start_time | grep '[r]sync' | awk '{print $1}'))
if [[ "${#pids[@]}" -gt 50 ]]; then
   # I am not sure how many process you want to kill. 
   # Let say you want to kill the first one.
   kill -9 "${pids[0]}"
fi

    Note: the grep pattern is '[r]sync' instead of 'rsync' because it excludes the grep command itself.

    In case if you want to limit the total number of process to 50. You could do something like this:

    declare -a pids
pids=($(ps -u test --sort=start_time | grep '[r]sync' | awk '{print $1}'))
if [[ "${#pids[@]}" -gt 50 ]]; then
    # You want to kill all execess processes
    excess=$((${#pids[@]} - 50))
    echo "Excess processes: $excess"
    kill -9 "${pids[@]:0:$excess}"
fi