I'm writing a python UI in kivy to manage some remote machines with fabric. As I can't use fabric's parallel implementation on Windows 10 (see here), I was hoping to use parallel-ssh to actually perform the parallel remote operations. This issue seems to be caused by the interactions between the libraries, rather than an issue with any single one of them.
I've tried manually loading my private key as suggested here:
from fabric.api import execute
import pssh
from pssh.utils import load_private_key
hosts = ['192.168.0.2']
private_key = load_private_key('C:/Users/democracy/.ssh/id_rsa')
pssh_client = pssh.ParallelSSHClient(hosts, user='XXX', password='YYY', pkey=private_key)
output = pssh_client.run_command('whoami', sudo=True)
pssh_client.join(output)
for host in output:
for line in output[host]['stdout']:
print("Host %s - output: %s" % (host, line))
The above code results in the following backtrace:
Exception: Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
Traceback (most recent call last):
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1884, in _check_banner
buf = self.packetizer.readline(timeout)
File "C:\environments\democracy\lib\site-packages\paramiko\packet.py", line 331, in readline
buf += self._read_timeout(timeout)
File "C:\environments\democracy\lib\site-packages\paramiko\packet.py", line 485, in _read_timeout
x = self.__socket.recv(128)
File "C:\environments\democracy\lib\site-packages\gevent\_socket3.py", line 317, in recv
self._wait(self._read_event)
File "C:\environments\democracy\lib\site-packages\gevent\_socket3.py", line 144, in _wait
self.hub.wait(watcher)
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 630, in wait
result = waiter.get()
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 878, in get
return self.hub.switch()
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 609, in switch
return greenlet.switch(self)
gevent.hub.LoopExit: ('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1740, in run
self._check_banner()
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1888, in _check_banner
raise SSHException('Error reading SSH protocol banner' + str(e))
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
General SSH error - Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
The above code works if I import pssh before fabric. Unfortunately, it seems if I do this, any buttons on my kivy interface (which kick off any operations in a background thread) block forever on press. If I go to the console after a button press and send a keyboard interrupt, kivy stops blocking and begins cleanup, but executes the command from the button press before exiting. The stacktrace on sending this interrupt is below:
[INFO ] [Base ] Leaving application in progress...
Traceback (most recent call last):
File "machine_control_ui.py", line 7, in <module>
DemocracyControllerApp().run()
File "C:\environments\democracy\lib\site-packages\kivy\app.py", line 828, in run
runTouchApp()
File "C:\environments\democracy\lib\site-packages\kivy\base.py", line 504, in runTouchApp
EventLoop.window.mainloop()
File "C:\environments\democracy\lib\site-packages\kivy\core\window\window_sdl2.py", line 659, in mainloop
self._mainloop()
File "C:\environments\democracy\lib\site-packages\kivy\core\window\window_sdl2.py", line 405, in _mainloop
EventLoop.idle()
File "C:\environments\democracy\lib\site-packages\kivy\base.py", line 339, in idle
Clock.tick()
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 553, in tick
current = self.idle()
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 533, in idle
usleep(1000000 * sleeptime)
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 717, in usleep
_usleep(microseconds, self._sleep_obj)
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 395, in _usleep
_kernel32.WaitForSingleObject(obj, 0xffffffff)
KeyboardInterrupt
*** BUTTON PRESS OPERATION OUTPUTS HERE ***
```
Any insight into why this might be happening and how I can avoid it would be much appreciated. I could potentially investigate other parallel ssh solutions (although I imagine anything using paramiko would have the same issue), or manually kick off a thread per host to achieve the parallel operation otherwise (which probably has its own list of headaches), but I'd prefer to just use the parallel-ssh library if there's a workable solution.
I'm using parallel-ssh 0.92.2 on Python 3 and Windows 10.
From docs -
parallel-ssh uses gevent’s monkey patching to enable asynchronous use of the Python standard library’s network I/O.
Make sure that ParallelSSH imports come before any other imports in your code. Otherwise, patching may not be done before the standard library is loaded which will then cause ParallelSSH to block.
If you are seeing messages like This operation would block forever, this is the cause.
Monkey patching is only done for the clients under pssh.pssh_client and pssh.ssh_client for parallel and single host clients respectively.
New native library based clients under pssh.pssh2_client and pssh.ssh2_client do not perform monkey patching and are an option if monkey patching is not suitable. These clients will become the default in a future major release - 2.0.0.
Since monkey patching is used for the client you are using, other uses of the threading, socket etc modules in your application will also have been patched to use gevent which means they no longer run in a native thread but in a co-routine/greenlet.
This is the reason your background thread operations block as they run in a greenlet on the same thread rather than a new thread.
As of 1.2.0, a new client based on libssh2 instead of paramiko is available which does not use monkey patching:
from pssh.pssh2_client import ParallelSSHClient
<..>
Rest of your application can then use the standard library as-is.