Search code examples
pythonpython-3.xsshsubprocesspsutil

subprocess and psutil : how to get a list out of subprocess.check_output?

I am willing to determine the current CPU usage of a bunch of Linux computers, all available via SSH connexion. As for now I have managed something via the psutil and subprocess modules, but it could be much more straightforward if I could get a list out of subprocess.check_output.

So I defined a cpu_script.py :

import psutil

print(psutil.cpu_percent(interval=1,percpu=True))

to be called in my main script, which is in the same folder, with:

import subprocess
import os
import numpy as np
import psutil

usr = "AA"
computer = "BB"
cpu_script = os.path.join(os.getcwd(),"cpu_script.py")
test = subprocess.check_output("ssh -X " + usr + "@" + computer + 
         "python3 -u - < " + cpu_script,shell=True).strip()

As cpu_percent's outputs are typically in a list, I would expect my "test" variable to be a list, but what I really get is a byte:

input(test)
>> b'[4.0, 5.0, 7.0, 10.9, 18.4, 1.0, 4.0, 3.0]'
input(type(test))
>> <class 'bytes'>

Here I have managed to bend everything to finally get an array (with decode + replace + np.fromstring), but it is far from satisfying:

tmp = test.decode("utf-8")
input(tmp)
>> [4.0, 5.0, 7.0, 10.9, 18.4, 1.0, 4.0, 3.0]
input(type(tmp))
>> <class 'str'>

tmp = (tmp.replace('[','')).replace(']','')
result = np.fromstring(tmp, sep=',')
input(result)
>> [ 4.   5.   7.  10.9 18.4  1.   4.   3. ]
input(type(result))
>> <class 'numpy.ndarray'>

Could'nt I have straightaway a subprocess.check_output variable that is a list or an array ? What am I missing ?

Also if there is a way to avoid defining a cpu_script, hence to pass directly the python command in the subprocess, I would be interested ! I have failed at all my attempts to do so for now, the tricky part here being that there is both a ssh connexion and a python3 console command followed by python commands.

Thanks for your help!

Solution

  • The underlying python program "serialized" the output as text.

    check_output returns a bytes object that you need to decode, then "unserialize".

    I suggest:

    import ast

result = ast.literal_eval(test.decode())

    The best way would be to call this sub-python script as a module. You wouldn't have to run a subprocess/serialize/unserialize in that case. Just import the sub-module, call a function and get a real list as a result. Of course, in the case of ssh subprocess call, you cannot.

    However you could improve this ugly and code-injection prone/unsafe check_output call needing shell=True by passing the filehandle of the opened python file to execute, feeding the output to the input of check_output, and using a list of strings instead of a string as argument:

    with open(cpu_script,"rb") as f:
    test = subprocess.check_output(["ssh","-X",usr + "@" + computer,"python3","-u","-"],stdin=f)

    followed by the decode+evaluation code.