I am finding it very difficult to wrap my head round this.
I am trying to accomplish something very simple which is to periodically copy files from one linux system to another through a script on local machine.
The remote system is very limited on memory and as such doesn't support cron or many of the common libraries. Only way is to run a script forever which I'd rather not do. It does have scp on it and ssh to connect to it
I want to ssh from my local machine to remote machine, run find command for all files less than a day old.
I want to then use this result and process it in the form of
scp someone@somewhere.com:"file1.sh file2.txt file3.jpg" /destination
destination being local location. It would be even better if I could rename the file to include full path of file as its name on the destination folder.
Edit 1:
Some progress I used the following command to get a list of files that I want to copy
ssh root@192.168.0.3 'cd /tmp && find -name *.mp4 -mtime -1'
I do have to manually enter password though so this can run as part of a script
To perform the transfer without entering a password, you have to decide which machine should trust the other. If the two hosts are SRC (host with the files) and DEST (host where the copies will go), then either SRC must trust DEST, or DEST must trust SRC (or both!).
Let's say that SRC will trust DEST but we don't want DEST to trust SRC.
To connect and run the copy without user interaction, you will have to store on DEST the equivalent of an unencrypted password. For obvious reasons, this is not a good idea but there are ways to minimise the risk.
One option is to install a command like sshpass but then you really will need to leave the unencrypted password lying around, so I recommend not using this method unless you really have to.
A better method is to set up public-key authentication between the two machines. Normally with this, you generate a private key which is stored encrypted. You type a passphrase to unlock it when you wish to use it, or you unlock it once and load it into a program like ssh-agent. The latter is probably the preferrable approach (the key is stored unencrypted in memory but an attacker has to have already obtained root access in order to make use of it. Clearly someone with root access could just record you typing in your password).
We can avoid the initial passphrase entry by storing the private key unencrypted. If we don't take special precautions, obviously having a private key stored unencrypted on DEST is equivalent to leaving the password to SRC lying around. This is bad. To minimise the risk, we can configure SRC so that it doesn't allow DEST to connect and run arbitrary commands. We can specify that only one command (let's call it COMMAND) is to be allowed. If we are careful, this mitigates the risks somewhat.
So, we've decided to configure DEST to be able to run COMMAND on SRC without any password entry. As we are on linux, we are probably using openssh. Here's a way to set things up:
dest:~$ cd ~/.ssh
dest:~/.ssh$ ssh-keygen -b 2048 -t rsa -N '' -f src-cmd
dest:~/.ssh$ ( echo -n 'command="/home/usr/mkfilelist" '; cat src-cmd.pub ) |\
ssh usr@src 'mkdir -p .ssh; cat >> .ssh/authorized_keys'
/home/usr/mkfilelist on SRC (if it exists!) without being asked for a password:dest$ ssh usr@src -i ~/.ssh/src-cmd
For even more security, consider using from=DEST,command=... so that only connections from DEST are allowed. See: authorized_keys(5) for the gory details.
Now we need to write /home/usr/mkfilelist.
These days, rsync is a good program for synchronising files between two sets of folders but I'm going to assume SRC provides that staple of historical UNIX: cpio!
cpio reads a list of filenames on stdin and outputs an "archive", or reads an "archive" and creates the files. This will work very well with your find command:
#!/bin/bash
cd /PATH/TO/SRCFILES
find . -mtime -1 -name '*.mp4' -type f -print | cpio -o
The important thing to note is that this will spew the "archive" to stdout (so don't try running it unless you redirect into a file). Also note that you should ensure that find does not output any filepaths that start with / (hence the cd and use of .).
Modify your find criteria to taste and save as /home/usr/mkfilelist on SRC.
Make it executable (chmod +x /home/usr/mkfilelist).
Now we can take these pieces and put together a script you can run from cron on DEST:
#!/bin/bash
cd /PATH/TO/DESTFILE
ssh usr@src -i ~/.ssh/src-cmd | cpio -i
This connects up to SRC, causing the "archive" to be generated. This is piped into a cpio running in an appropriate folder on DEST which extracts the files.