Python - make_archive zip in multiprocess pool not working properly

Question

I want to create a zip archive from multiple processes at the same time. When I've used pool from multiprocessing. Some of zip archive have not been created. It seems that the close method do not wait for processes to end Below the process launcher:

import os
import sys
import json
from multiprocessing import Pool
from shutil import copyfile, make_archive
from xml.etree.ElementTree import ElementTree

def launch_traitment(path, path_upload):
    print(path)
    with Pool(processes=NB_PROCESS) as pool:
        for dir in os.listdir(path):
            pool.apply_async(compute_dir,(dir,path,path_upload,))
        pool.close()
        pool.join()
...
def compute_dir(dir, path, path_upload):

    working_path = path+'/'+dir
    deleteAck(working_path+'/'+dir+'.ack')
    execute(dir, path)
    generateZIP(dir, working_path, path_upload)
...
def generateZIP(dir, working_path, path_upload):
    lst_file_meta_data = dir.split('_')
    if len(lst_file_meta_data) < 3:
        print(f"File {dir} incorrect naming")
        return 1

    provider = lst_file_meta_data[0]
    registration = lst_file_meta_data[1]
    session_date = lst_file_meta_data[2]

    zip_file = path_upload+'/'+provider+'/'+registration

    if not os.path.exists(zip_file+'/'+ dir +'.zip'):
        print('Génération du ZIP : ', zip_file+'/'+ dir +'.zip')
        if not os.path.exists(zip_file):
            os.makedirs(zip_file)
        make_archive(zip_file+'/'+ dir, 'zip', working_path)

I've tried to create zip with the system command but I had the same problem:

os.system(f'zip -r -j {zip_file}/{dir}.zip {working_path}')

I've also tried to put a try except but no exception has been thrown

Answer 1

I've found the answer. I've a number of processes lower than the number of processes launched by my for loop. So if the process have not finished his treatment, it's overrided by a new one.