Popen fails to start a process when I call it from a python script that runs as a Linux service

Question

I have a script which on certain conditions calls Popen to create new process. If I run this script from the command line, then Popen creates a process. But if I run the same script as a linux systemd service, then script executes normally except Popen part which fails to start a process. Same Popen failure thing happens if I start this script from within Java application deployed and running in Tomcat

I've tried to mix arguments in Popen construct, but I feel that I am missing something about the whole idea behind linux processes relationships Here is the code which executes perfectly if started from cmd terminal like this: "python3 myscript.py" and fails if run as systemd service( systemctl start myscript.service - config file ofcourse exists) Fails Popen part

Service config file:

[Unit]
Description=Some Service
After=multi-user.target
Conflicts=getty@tty1.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/bin/my_service.py
StandardInput=tty-force

[Install]
WantedBy=multi-user.target

Python script my_service.py itself:

import socket, os, signal, psutil
from subprocess import Popen, PIPE, DEVNULL

def find_free_port():
    s = socket.socket()
    # Bind to a free port provided by the host.
    s.bind(('', 0))
    return s.getsockname()[1]

HOST = '91.230.195.104'
PORT = 65440

lastProcess = None
currentPID = os.getpid()

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:

   s.bind((HOST, PORT))
   s.listen()

   while True:
       try:
           print('Waiting for new Connection..')

           conn, addr = s.accept()

           if lastProcess != None:
               current_process = psutil.Process(lastProcess.pid)
               children = current_process.children(recursive=True)
               for child in children:
                   if child.pid != currentPID:
                       os.kill(child.pid, signal.SIGKILL)

           print('\nConnected to: '+str(addr))

           port_sender = find_free_port()
           port_videolink = None

           while True:
               port_videolink = find_free_port()
               if port_sender != port_videolink:
                   break

           with conn:
                data = conn.recv(1024)
                print('DRONE ID: '+str(data.decode()))
                if data:
                    conn.sendall(bytes(str(port_sender)+':'+str(port_videolink), 'utf-8'))
                    lastProcess = Popen("python3 video_receiver.py --port_videolink="+str(port_videolink)+
                                                       " --port_sender="+str(port_sender), shell=True, preexec_fn=os.setpgrp, stdout=PIPE, stderr=PIPE)
                    print("Videofeed port: "+str(port_videolink)+"\n")
       except Exception as e:
             print(str(e))

there are no error or warning, just video_receiver.py process never starts :-\

Answer 1

The issue is that the CWD for the process is different. To debug this have your program log the result of os.getcwd(). Another common issue although not the one occurring here is that PATH is different.

lastProcess = Popen("python3 video_receiver.py")

fails because os.path.isfile(os.path.join(os.getcwd(), "video_receiver.py")) == False

One solution is to provide the full paths to python and your script in the Popen call. The path to Python is only needed if you have Python installed in a virtual environment.

Another is to use os.chdir() or set the working directory in the service configuration.