ssh, nohup, then copy results back

I want a script which automates the running of big mathematical computing jobs on a remote machine. Currently I do this:

• ssh into remote machine; run matlab script
• periodically check in to see if job is done yet
• when job is done, I manually scp the files containing the results back to my home machine.

I tried to use these kind of lines in my script to do this job (note the script runs on my machine):

ssh nohup matlab -r theScript; exit;
scp remote@~/files ~/files

This doesn't work. After a while the ssh session ends and the script just proceeds to do the scp, even though the job is not finished yet and the files don't exist yet.

I think what I need to do is check periodically whether the job is done, perhaps by ssh'ing in periodically and reading the nohup.out file looking for a DONE! signal using grep. Then when I see that, copy the files back. But this seems com开发者_C百科plicated and I don't know how to get the DONE! message back to my computer to run a conditional on (if you see the DONE signal, do this stuff...) any ideas?

Yeah, the ssh could time out or something. So yeah, you best option is to poll. e.g.

RUN="$(date +%Y%m%d-%H%M%S)"
ssh remote " nohup bash -c \"( matlab -r theScript; echo \$? > $RUN.done ) >$RUN.log 2>&1 </dev/null &\" "
DONE=""
while [ -z "$DONE" ]
do
    sleep 60
    DONE="$(ssh cat $RUN.done 2>/dev/null)"
done
if [ $DONE -eq 0 ]
then
    scp ...
else
    # Optionally fetch logfile
    # scp $RUN.log@remote ...
    echo "ERROR in remote matlab...."
fi

There seems to be a way to configure this in your ssh_config.

This page explain how: http://nileshbansal.blogspot.com/2007/02/prevent-timeouts-in-ssh.html

Try setting the ServerAliveInterval in you ssh_config file. That way, your SSH session will not close until the remote command finishes.

See here.