I've tried threading an issue to where I log into multiple devices, and log data. I've read several articles into threading for Netmiko, and I can't seem to get the threading to work. Tje code calls other classes from the main script. here is an example with threading, it comes out to the same 18 seconds with or without threading. Network devices use I/O operations, as does this code, so threading is preferred over pools.
thread_pool = Queue()
Hardware = HardwareMaintenance.hardware_handler
ip_pool = Queue()
function_list = []
thread_list = []
class Connect:
def __init__(self,):
intake_file = open('ip_list.txt', 'r')
self.json_data = [json.loads(line) for line in intake_file]
pass
def connect_parser(self, ):
for data in self.json_data:
ip = data["ip"]
ip_pool.put(ip)
username = data["username"] if data["username"] else ""
password = data["password"] if data["password"] else ""
secret = data["secret"] if "secret" in data else False
device_type = data["device_type"] if data["device_type"] else ""
if data["header"] == "Netmiko":
print("The variables being passed: " + ip, username, password, device_type)
ConnectHandler = netmiko.ConnectHandler(
device_type=device_type,
host=ip,
username=username,
password=password,
port=22,
secret=data["secret"] if "secret" in data else False)
if ConnectHandler:
try:
ConnectHandler.enable()
except Exception as e:
print("Could not connect to {}".format(ip))
thread_pool.put(ConnectHandler)
conn = thread_pool.get(block=False)
host = ip_pool.get(block=False)
res = threading.Thread(target=Hardware, args = (conn, host), daemon=True)
res.start()
res.join()
if __name__ == "__main__":
functions = {'Hardware': Hardware}
from main import Connect
Connector = Connect()
Connector.connect_parser()
I tried context managers under main and the same time always executes.
I'm not sure if there's something I'm missing or if I'm using threading correctly, any suggestions would be appreciated.
I'm thinking maybe 18 seconds is good time to begin with ~ however I've read articles working with threading and network devices and their time seems to significantly improve using threading.
Okay so I got it to work by unindenting the join, putting the threads in a list first after they were started, then joining them, instead of joinging them as they were being executed, because i think by doing that I wasn't actually joining them. I'm still not clear as to why a context manager wouldn't work under main. If any one has insight please let me know, thank you. This code took it from 18 seconds down to 12. This is a significant improvement if I were running the code against hundereds or more devices, and calling more classes. Notice the last few lines of code below it got it to work for me.
def looper():
while not thread_pool.empty():
conn = thread_pool.get(block=False)
host = ip_pool.get(block=False)
print("host " + host)
res = threading.Thread(target=Hardware, args = (conn, host), daemon=True)
res.start()
print(res.__repr__())
thread_list.append(res)
#main_thread = threading.current_thread()
#main_thread = current_thread()
for i in thread_list:
i.join()
print("This is the threads list")
print(i.__repr__())
if __name__ == "__main__":
from main import Connect
Connector = Connect()
print(Connector)
print(time.perf_counter())
Connector.connect_parser()
# with ThreadPoolExecutor(max_workers=8) as exe:
# exe.submit(Connector.connect_parser())
print(time.perf_counter())
Your problem is that you wait for the thread to finish after you started it. This logic is basically sequential, not parallel. You should first create and start all threads, save them somewhere and wait for them AFTER all threads were started. Consider the following code:
# Create and run threads
threads = []
for data in self.json_data:
# prepare your connect handler...
thread = threading.Thread(..., daemon=True)
thread.start()
threads.append(thread)
# Wait for all threads to complete
for thread in threads:
thread.join()