ssh, nohup, then copy results back

I want a script which automates the running of big mathematical computing jobs on a remote machine. Currently I do this:

ssh into remote machine; run matlab script
periodically check in to see if job is done yet
when job is done, I manually scp the files containing the results back to my home machine.

I tried to use these kind of lines in my script to do this job (note the script runs on my machine):

ssh nohup matlab -r theScript; exit;
scp remote@~/files ~/files

This doesn't work. After a while the ssh session ends and the script just proceeds to do the scp, even though the job is not finished yet and the files don't exist yet.

I think what I need to do is check periodically whether the job is done, perhaps by ssh'ing in periodically and reading the nohup.out file looking for a DONE! signal using grep. Then when I see that, copy the files back. But this seems complicated and I don't know how to get the DONE! message back to my computer to run a conditional on (if you see the DONE signal, do this stuff...) any ideas?

Solution

Yeah, the ssh could time out or something. So yeah, you best option is to poll. e.g.

RUN="$(date +%Y%m%d-%H%M%S)"
ssh remote " nohup bash -c \"( matlab -r theScript; echo \$? > $RUN.done ) >$RUN.log 2>&1 </dev/null &\" "
DONE=""
while [ -z "$DONE" ]
do
    sleep 60
    DONE="$(ssh cat $RUN.done 2>/dev/null)"
done
if [ $DONE -eq 0 ]
then
    scp ...
else
    # Optionally fetch logfile
    # scp $RUN.log@remote ...
    echo "ERROR in remote matlab...."
fi