Count ssh find results

Question

I have this part in my script where I would copy 1 hour old files from one server to the current server I am currently logging in:

ssh testuser@sampleServer find /home/testuser/files -type f -mmin -60 > /home/user/bin/tmp/results.txt

while read LINE
do
    scp -r testuser@sampleServer:$LINE /home/user/bin/tmp
done < /home/user/bin/tmp/results.txt

I want to count the number of files retrieved in the first line of the code and store its value in a variable, so I just put:

COUNT=`ssh testuser@sampleServer find /home/testuser/files -type f -mmin -60 | wc -l`

But I believe this can still be improved, since it is redundant. Do you have any suggestions as to how I can revise it? Thanks

Answer 1

Opening n+1 TCP connections and performing SSH handshakes etc for each to copy n files seems horribly inefficient.

This is how you would usually do it (and also avoid that pesky temporary file):

ssh testuser@sampleServer '
    find /home/testuser/files -type f -mmin -60 -print0 |
    xargs -r0 tar zcf -' |
tar -C /home/user/bin/tmp -z -x -f -

This is assuming there are no newlines in any file names (which would have broken your original script as well) and that you don't mind creating a mirror directory structure in /home/user/bin/tmp if there is one on the remote site.

Now, adding a line count to your original script would have been simple -- why not run it on the temporary file?? -- but it can be similarly easy here. Just add a -v option to the extracting tar filter and cound the number of output lines.

... | tar -C /home/user/bin/tmp -z -x -v -f - | wc -l