I've written a computer vision script that requires the use of a GPU, but my main webserver doesn't have a GPU (and can't have one).
As such my main webserver is box A, and I've bought a separate server, box B, with a GPU.
Workflow:
1) Users of my site upload photos to box A.
2) I process them for object-detection on box B (in batches as the object-detection code has a slow initial start-up)
3) And then I store the results on box A.
Currently I have 3 cron scripts:
1) box A: script makes filename list of most recent un-processed 1000 files.
(This runs every 5 minutes and typically takes a second.)
2) box B: using rsync, the script retrieves filename list from box A, then retrieves all of the files in the list, runs Yolo object detection on the list of images, and saves the results to results.txt
(This runs every 5 minutes, 1 minute after script 1 and typically takes 2 minutes.)
3) box A: another script connects to box B, retrieves results.txt, and processes them. Images are marked as processed so that the next time script (1) runs it doesn't do the same files again.
(This runs every 5 minutes, 1 minute before script 1 and typically takes a few seconds.)
What I'd like:
In theory I feel I could probably run all of this from a single script on box A, which would be cleaner and simpler. I'm concerned about what'll happen when one of the scripts gets delayed. I could write logic in to add some checks (touch script2_completed, for example), but I think I should consolidate all of the logic into one script, that executes code on both servers.
My ignorance:
I've never done this before though, and haven't a clue how best to go about it, and what's possible/isn't possible.
For instance, if I start running the remote object detection script on box 2, will the script running on box 1 wait until it's finished before it moves on to the next command, or would it end up being asynchronous, which would be undesirable?
Many thanks!
Turns out this is ridiculously simple.
To execute a script on the remote machine from the local machine, it's as simple as
ssh user@serverIP sh /run/this/script.sh
And as @Aaron says in a comment above, that runs as a blocking request so your local script won't resume until that remote script has finished.