Search code examples
pythonmultiprocessingreturn

How to have 2 different function running parallel that return an array in Python


So I have to make a program that searches a keyword through files contained in folder and prints number of times it's used in one. To reduce the search time I've decided to break the folder in 2 folders and have two separate functions that run in parallel, each searches through different folder.

Here is one of the two functions that searches through the files (the other is the same):

    pathText1 = '/Papers/scripts1'
    countmatch1 = 0;
    matrix1 = [[]]
    for filename1 in os.listdir(pathText1):
        fileDir1 = pathText1 + '/' + filename1
        fileText1 = open(fileDir1, "r",encoding='utf8')
        content1 = fileText.read()
        content1 = content1.lower()
        countn1 = content1.count(keyword)
        if count1 > 5:
            print ('The word:   ' + keyword + '    | was found ' + str(count1) + ' times in the file: ' + filename1)
            countmatch1 = countmatch1 + 1
            matrix1.append([filename1,count1])

    print('found ' + countmatch1 + ' matches')
    del matrix1[0]
    return matrix1

So, now I have a problem on how to implement the multiprocessing and make it have the two function return the matrix to the main. Thanks in advance for your help!


Solution

  • You should use shared variable. Check the example:

    from multiprocessing import Process, Manager
    
    def your_function(first_param, second_param):
        # Your work here to search keywords
    
    if __name__ == '__main__':
        manager = Manager()
    
        # shared variable
        matrices = manager.list()
    
        jobs = []
    
        # Range defines how many times you want to call the method.
        for each_process in range(3):
            process = Process(target=your_function, args=(first_param, second_param))
            jobs.append(process)
            process.start()
    
        for each_process in jobs:
            each_process.join()
    
        print(matrices)
    

    Also, you can check a more detailed tutorial from here.