Search code examples
pythonpython-multiprocessing

Is this a wise use of a global variable with multiprocessing?


I'm using Python's multiprocessing library to send a bunch of data to an API. I have to send one record per request, so I'm using pooling to make n parallel requests. I'd like to take results from those requests (summarized into a string) and combine it with other data for logging.

So here is a simplified example of what I'm trying, distilled down to the multiprocessing itself. Is it wise to use a global variable like this? Is there a better way?

Thanks

import multiprocessing

message = ""

def cTest(m):
    global message
    message = "-".join(m)

def mTest(name):
    print(name)
    return("t" + name)


if __name__ == '__main__':

    testList = ['Jake', 'Jake', 'Jake', 'Jill', 'Jake', 'Jake']

    pool = multiprocessing.Pool(processes=2)
    pool.map_async(mTest, testList, 1, cTest)
    pool.close()
    pool.join()

    print("-->" + message)

Output

Jake
Jake
Jake
Jill
JakeJake

-->tJake-tJake-tJake-tJill-tJake-tJake

Solution

  • I would do this without cTemp and global

    results = pool.map_async(...)
    
    values = results.get()
    
    message = "-".join(values)
    
    print("-->" + message)
    

    EDIT:

    Full working code

    import multiprocessing
    
    def mTest(name):
        return "t" + name
    
    if __name__ == '__main__':
    
        testList = ['Jake', 'Jake', 'Jake', 'Jill', 'Jake', 'Jake']
    
        pool = multiprocessing.Pool(processes=2)
    
        results = pool.map_async((mTest, testList, 1)
    
        pool.close()
        pool.join()
        
        values = results.get()
        message = "-".join(values)
    
        print("-->" + message)