Run a script on remote host without caring ssh session

I have a python script in a folder of a remote machine. For executing it, I make a ssh session from my local computer, go to that folder and run.

Commands I use:

ssh remotehost
user@remotehost:~$ cd /my/folder
user@remotehost:~$ python abc.py >> abc.log

Now the problem is ssh session. The script takes lot of time and due to internet issues, ssh session terminates and script doesn't complete. Assume that the remote is always up and running.

Can I run the script without caring of the ssh session termination and do tail -f abc.log with ssh anytime I want ?

Solution

You can run the script either in a screen or can run the process in nohup+bg. I always prefer Screen but let me explain both methods.

1. nohup

You can use nohup command to run a process by detaching from terminal like this nohup python /my/folder/abc.py & This by default creates nohup.out file where all the logs will be stored. If you want custom file then you can use redirection then it will be nohup python /my/folder/abc.py >> abc.log &

In single command it will be

ssh user@remotehost 'nohup python /my/folder/abc.py >> abc.log &'

nohup wikipedia

2. Screen

From the docs.

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. Each virtual terminal provides the functions of the DEC VT100 terminal and, in addition, several control functions from the ISO 6429 (ECMA 48, ANSI X3.64) and ISO 2022 standards (e.g. insert/delete line and support for multiple character sets). There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows the user to move text regions between windows.

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user's terminal.

Screen Manual

So you can directly run the script in screen using ssh and then you can view the logs whenever needed either by attaching to screen or you can redirect the logs to some file directly or redirect to both file and out put using tee.

Run command in screen and print output in stdout(terminal).

ssh user@remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py; exec bash")'

Run command in screen and redirect output to file.

ssh user@remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py >> abc.log &2>1; exec bash")'

Run command in screen and redirect output to both file as well as to stdout(terminal).

ssh user@remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py &2>1 |tee abc.log; exec bash")'

Note: In all the above commands exec bash is needed else the screen will terminate once the job complets.

Any one of the above commansd should do the job.
In all the above cases you can attach the screen ScreenName using screen -r ScreenName and can see the logs.
I always recommend stderr redirection when redirecting to a file.

some references on using linux screen