concurrent.futures.ProcessPoolExecutor() map can't read global variable

Question

I have simple code bellow using netmiko for network automation: so first i have:

1. function cisco_command for read command from input file
2. Function cisco_host for read host info from input file
3. funtion open_connection for initiate connection to device
4. function run_program for multiprocessing
5. function main is main program

so I have problem with our code , you can see on open_connection func we have global variable as commands_info but if we run this program via multiprocessing (run_program func) the global variable can't read by open_connection func.

import time
import os
import concurrent.futures
from netmiko import ConnectHandler
from functools import partial


full_path = os.path.dirname(__file__)
host_file = os.path.join(full_path, "lab-router.txt")
command_file = os.path.join(full_path, "cisco-log.txt")
starting_time = ""


command_info = []

def cisco_command():
    global command_info
    command_info = []
    with open(command_file, 'r') as commands:
        for line in commands:
            com = line.strip()
            command_info.append(com)
    return command_info
    
def cisco_host():
    global starting_time
    hosts_info = []
    with open(host_file, 'r') as devices:
        for line in devices:
            deviceip = line.strip()
            host = {
                'device_type': 'cisco_ios',
                'ip': deviceip,
                'username': 'dodo',
                'password': 'dodo',
                'secret': 'dodo'
            }
            hosts_info.append(host)

    starting_time = time.perf_counter()
    return hosts_info

def open_connection(host):
    global command_info
    sendcommand = ""     
    try:
        connection = ConnectHandler(**host)
        print('Connection Established to Host:', host['ip'])
        connection.enable()
        for i in command_info:
            sendcommand += "\n"
            sendcommand += "==== {} ====".format(i)
            sendcommand += "\n"
            sendcommand += connection.send_command(i)
            sendcommand += "\n"
# return sendcommand
        with open("{}/{}_log.txt".format(full_path, host['ip']), 'w') as nf:
            nf.write(sendcommand)  
    except:
        print('Connection Failed to host', host['ip'])
    

        
def run_program(hosts_info):
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = executor.map(open_connection, hosts_info)
            
        for result in results:
            pass

        finish = time.perf_counter()
        print('Time Elapsed:', finish - starting_time)
    
def main():
    commads = cisco_command()
    hosts_info = cisco_host()
    run_program(hosts_info)


    
if __name__ == '__main__':
    main()

I don't know what I wrong

Answer 1

You wrote:

if we run this program via multiprocessing (run_program func) the global variable can't read by open_connection func.

This statement is actually incorrect. Your global variable command_info is indeed read by the function open_connection. This is not the issue.

The issue I see you are encountering is that you are trying to use the command global to allow you to update the changes made in command_info processed by each cpu core running the open_connection function to the global command_info in your main core. This won't work. Imagine you have 4 cpus running concurrently and each cpu trying to amend the same global term at the same time. That will be a nightmare scenario if Python allowed that which it does not allow.

In actuality, Python allows you to pass command_info into each CPU core (w/o using global) and you can think of that command_info as now a global variable unique to that compute core. To pass the updated command_info of each unique core back to the master core that is running concurrent.futures.ProcessPoolExecutor(), you have to return command_info at the end of the function open_connection(host) (along with your sendcommand). Then in the main core, you can access it as one of the terms in result of results. Thereafter, you can update the global command_info variable in your main core.

Hope this explanation helps you understand your issue. :)